Frank Sauerburger

Vibe code in Rust

2026-04-29T00:00:00+02:00

If you are vibe coding and you don’t review any of the code, like a true vibe coder, you should build your system in Rust. I know, that’s a controversial statement. But hear me out. We will see how good the advice is.

Why?

Letting AI write your code in Rust has a couple of advantages:

Better code quality. Rust’s strict compiler and ownership model can help prevent common bugs and memory issues, leading to more robust code. If it compiles, you have some assurance it will not crash on Saturday at 3 am. No AttributeError: 'NoneType' object has no attribute 'something' or whatever.
Performance. Before the advent of good Generative AI, there was always a trade-off between speed of code and speed to code. Often, especially for early market tests, prototypes, or personal experiments, the speed of execution was not the bottleneck, but the speed of development was. When AI writes the code for you at lightning speed, the bottleneck shifts to defining the task and execution speed.

Why not?

There are also some arguments that speak against using Rust for vibe coding:

Less prevalent in training data. Rust became very popular in the last few years, but is still much less prevalent than Python or JavaScript/TypeScript. Therefore, AI models are commonly trained much less on Rust code than on other programming languages. Therefore, the quality of generated Rust code may be comparatively lower.
Language complexity. The Rust programming language is complex with a steep learning curve and novel memory management concepts. What’s difficult for human developers to learn and master is also difficult for AI models to generate correctly. This can lead to more errors during implementation.
Maintenance challenges. Due to the complexity of Rust, in the long term, it might be more difficult to maintain and review the generated code. However, since our initial premise was that we are not reviewing the code anyway, this point is moot.
Higher generation costs. The language complexity and the need for more iterations in AI thinking and iterations until the code compiles can lead to longer generation times and higher token costs.

Let’s put these things to the test and see if the positive or negative argument outweighs the other.

The experiment

I let Claude Code implement three different spec.md files, each in Python, Rust, and Typescript. So a total of 9 implementations. The spec files were identical across languages; only the language-specific instructions and conventions were adapted.

The experiment uses an automated framework that runs Claude Code in headless mode inside isolated Docker containers, one per language and project. Each container receives the same specification file and is tasked with producing a working implementation. The framework records three metrics for each run: the USD cost charged by the Claude API, the wall-time, and the correctness of the generated code verified via a shared test suite (which is also AI-generated, but reviewed to ensure applicability and correctness with the given specs).

Three language environments were tested:

Python 3.14 (using the uv package manager)
TypeScript (Node 25)
Rust (1.94 with Cargo)

Why these three? They are all very popular languages. All three are memory safe. Python is interpreted and dynamically typed. TypeScript is statically typed, and Node might use just-in-time compilation, while Rust is also statically typed and ahead-of-time compiled. The three languages are usually used for different purposes: Python is often used for data science, scripting, and web development; TypeScript is popular for frontend and backend web development; Rust is favored for systems programming and performance-critical applications. This gives us a good variety of language features and paradigms to compare.

The setup is documented at gitlab.sauerburger.com/vibe-code-language/utils.

The benchmark projects

Three projects of increasing complexity were chosen as benchmarks. The specs are available at gitlab.sauerburger.com/vibe-code-language/utils/-/tree/main/specs.

Mandelbrot: The simplest project. Claude had to implement an HTTP API that renders a blue-scale image of the Mandelbrot set for a given coordinate region, plus a single-page HTML/JS viewer with click-to-zoom navigation. Pure computation with no external dependencies or database.

Vocab Trainer: A mid-complexity CLI application for spaced-repetition vocabulary learning. The tool has two commands: search (look up a word via a free online dictionary API and persist it to SQLite) and train (quiz the user on stored definitions using a 6-bucket spaced-repetition scheduler). Requires file I/O, HTTP requests, and an embedded SQLite database.

Spending API: The most complex project. A production-oriented LLM usage-tracking system consisting of an HTTP API service backed by PostgreSQL and an admin CLI. It handles API-key authentication (Argon2id hashed secrets), per-request token consumption recording, cost computation with fixed-precision decimal arithmetic, and aggregated usage reporting grouped by model or API key.

The results

Feel free to review the 9 implementations yourself.

The following chart summarizes the API cost and generation time for each language and project. The numeric values can be found in the table at the end of this article.

Let’s break it down.

Cost: For the simple Mandelbrot project Rust was actually the cheapest to generate at $0.23, beating Python ($0.38) and TypeScript ($0.35). However, that advantage reverses as complexity grows: for the Vocab Trainer Rust already costs more ($0.87) than Python ($0.80) and TypeScript ($0.72), and for the Spending API Rust was the most expensive at $1.31, 42% more than Python ($0.92) and 30% more than TypeScript ($1.01).
Time: Rust took the longest to generate across all three projects by a wide margin, roughly twice the time of Python and 50–70% more than TypeScript. However, the time difference might also stem from the toolchain and environment setup. For example, loading all the Rust dependencies and compiling them into a binary, usually takes longer than installing Python packages or Node modules.
Correctness: All three languages passed the Vocab and Mandelbrot test suites completely. The Spending API is where things diverged: Rust passed 23 out of 24 test groups (failing only 1), Python passed 21 out of 24 (failing 3). The test failures are due to different returned HTTP status codes (400 vs 422) for invalid input, which is a minor correctness issue. In contrast, the TypeScript version could not even run the database migration and therefore produced no passing results at all.
Runtime performance: The Mandebrot project is purely CPU-bound and serves as a test-bed for CPU performance. We measure the time it takes to generate one image. Rust and TypeScript matched each other at ~0.4 s for the Mandelbrot benchmark, while Python was 3.5× slower at 1.4 s.

The conclusion

The hypothesis that Rust’s compile-time checks help vibe coding partially holds. Rust produced the most correct Spending API implementation and never crashed. But it comes at a cost: generation takes longer and is more expensive for complex projects. Python is the cheapest and fastest to generate and was surprisingly competitive on correctness. TypeScript was middle-ground on cost, but its generated code failed hardest on the most complex project.

So, should you vibe code in Rust? As always, it depends. It depends on your priorities: cost vs execution speed and stability.

The choice of language should depend on your application and your capabilities to review and maintain generated code. Let’s be honest, not reviewing any of the generated code is a bad idea.

Project	Language	Cost (USD)	Time (s)
Mandelbrot	Python	$0.38	127
Mandelbrot	TypeScript	$0.35	166
Mandelbrot	Rust	$0.23	323
Vocab Trainer	Python	$0.80	171
Vocab Trainer	TypeScript	$0.72	301
Vocab Trainer	Rust	$0.87	526
Spending API	Python	$0.92	367
Spending API	TypeScript	$1.01	422
Spending API	Rust	$1.31	771

Claude Code as member of the engineering team

2026-03-03T00:00:00+01:00

I made the experiment: Claude Code as a full engineer on GitLab, as part of the development team for a mini game. I create tickets and review merge requests, while Claude Code submits code as merge requests and iterates on them if required. It worked surprisingly well.

What’s the project? HTTP is the default protocol we use to connect with people and look up information on the web. For a long time, I wanted to offer a service accessible via SSH, where the terminal replaces the browser and SSH replaces HTTPS. To achieve that, I wanted to build a small terminal-based game. Voilà: AsciiMoria. An SSH-based game where players navigate their way through deep and dangerous mines, implemented in Rust.

The development workflow

Claude Code was wired into a project on GitLab, waiting to be assigned to issues with instructions on what to implement. Once assigned, Claude would work on a feature branch and interact with the ticket, asking for clarification if needed, for example. Each ticket would eventually result in a merge request for me to review. If I requested changes, or if the CI/CD pipeline failed, Claude would give it another shot and improve on the previous implementation.

What I liked most about this workflow was remaining in charge. I create tickets, write specs, review merge requests, request changes, and make decisions about the direction of the project. This meant I could focus on thinking about the game’s direction, corner cases, and new features without getting lost in the nitty-gritty details of building a game or an SSH server in Rust. The separation of concerns felt natural: I owned the vision, Claude owned the implementation.

The outcome

Coming straight to the point: the outcome of my experiment is a fun little game, free for everyone to play or fork.

To play it, you need a terminal and an SSH client with a local private key for authentication (ssh-keygen -t ed25519 if you don’t have a key already).

ssh asciimoria.com

The code is open source and available at https://gitlab.sauerburger.com/frank/asciimoria. The repository contains all interactions and conversations with Claude Code.

Is the game bug free? I don’t think so. Would it be bug free if I wrote the game? Also no. I made some contributions to prove to myself that I understand the code.

What did I learn?

A lot. I don’t want to sound foolish claiming that AI taught me things, but I freely admit I picked up many tricks along the way: how to approach certain patterns in Rust, the landscape of open source Rust libraries, new(-ish) features in Docker Compose files, new(-ish) features in GitLab CI/CD configs. Claude is extremely versatile and proficient across a remarkable breadth of technologies.

I also learned just how good AI is at getting things right the first time. We have all committed endless commit chains to fix a CI problem, caused by stupid mistakes, syntax errors, typos, or a general lack of understanding of how to configure the pipeline. Claude did none of that. In nearly every case, it had things right on the first or second attempt.

Perhaps the most surprising lesson was about the level of detail in tickets. When I was thorough in the issue description, Claude followed it to the letter - it had all the specification it needed and delivered exactly what I asked for. On the other end, when tickets were brief and superficial, Claude found sensible ways to fill in the gaps, and I had very few bad surprises. However, there is a dangerous middle ground. When I tried to be clever in the ticket description without fully understanding the underlying complexity and tricky details, Claude would take me at my word and implement something rather nonsensical. To be clear, Claude did see the hidden complexity, it simply trusted that I had made a deliberate choice when writing the specs. The lesson: be either very detailed and understand the underlying complexity or be very brief. Vague specificity is the worst of both worlds.

How was the experience?

Creating this game with AI feels a bit like cheating.

On the other hand, it also gave a very rewarding feeling. Closing tickets and burning through your backlog in very little time releases a lot of dopamine. Going back to writing code the hard way afterwards feels a bit like going through detox.

I was incredibly fascinated by how skilled my new engineer was. And I am not going to lie: it was very convenient to write tickets from my phone on the train and have a merge request ready by the time I got home, with passing unittests and the desired feature materialized in code.

However, I can see people trying to keep an AI agent working 24/7, which is not going to be healthy for our work-life balance. Just because you can create tickets (or maybe even review code) on your phone does not mean you should.

Where to go from here

For hobby projects, there is one important question to ask: do you prefer the activity of coding, or do you prefer having the finished project? To code vs the code.

Code generation has become very cheap, but the cost of ownership remains. You still have to understand the code you ship, maintain it, and make architectural decisions that go beyond what any AI can infer from a ticket. Even before good code generation was available, a key skill in software engineering was to focus on outcome rather than output. That principle is more relevant than ever, now that generating code is just a ticket away. The real work, deciding what to build and why, has not changed.

Async database access in Rust with Diesel 101

2026-02-08T00:00:00+01:00

Diesel, a database ORM in Rust, provides compile-time type checks for its database operations, seamlessly integrating with blazingly fast, robust API applications built with Axum. As always, when working with a new technology, some of the details are difficult to remember (assuming you’re not vibe coding the entire app). For example, is it load(&mut conn) or execute(&mut conn) for a select query? This article showcases a few simple queries. It doesn’t try to be a comprehensive tutorial. In the following, upper-case text is meant as a placeholder. The code assumes an auto-generated schema.rs file and a hand-written models.rs file with Rust representations of database data structures.

Select

let values: Vec<SELECTED_TYPE> = schema::TABLE::table
  // .inner_join(schema::OTHER_TABLE::table)
  .filter(schema::TABLE::FIELD.eq(VALUE))
  .select(schema::TABLE::FIELD)
  // or   .select((schema::TABLE::FIELD, …) )
  // or   .select(model::TABLE.as_select())
  // .offset(OFFSET)
  // .limit(LIMIT)
  .load(&mut conn)
  // .first(&mut conn)
  .await?;

Insert

let values: SELECTED_TYPE = VALUES_OBJECT
  .insert_into(schema::TABLE::table)
  .returning(schema::TABLE::FIELD)
  .get_result(&mut conn)
  // or without returning
  // .execute(&mut conn) 
  .await?;

Update

diesel::update(schema::TABLE::table)
    .filter(schema::TABLE::FIELD.eq(VALUE))
    .set(schema::TABLE::FIELD.eq(VALUE))
    .execute(&mut conn)
    .await?

Virt-install a Ubuntu VM from the terminal

2026-01-03T00:00:00+01:00

In recent releases of Ubuntu, the installation experience has changed significantly. Starting with Ubuntu 24.04, the default server ISO uses the new Subiquity-based live installer, and the traditional text-based “mini.iso” workflow is no longer provided in the same way.

If you’re provisioning virtual machines via automation — especially on a headless host using KVM and virt-install — this can be frustrating. By default, virt-install may attempt to open a graphical console, pushing you toward VNC or SPICE even when you prefer a pure terminal-based workflow. If you open the virt-install console, you’ll see something like:

WARNING CDROM media does not print to the text console by default, so you likely will not see text install output. You might want to use --location. See the man page for examples of using --location with CDROM media

This guide shows how to install Ubuntu 24.04 LTS using the live server ISO entirely from the terminal, launching the installer kernel and initrd directly from the ISO.

Define Virtual Machine Parameters

Start by defining the VM configuration. The block below is fully editable — adjust CPU, RAM, disk size, and paths to fit your environment before running it.

export VM_NAME="ubuntu-vm"
export VM_ISO="https://releases.ubuntu.com/24.04.3/ubuntu-24.04.3-live-server-amd64.iso"
export VM_OS="ubuntu-stable-latest"
export VM_IMG="/var/lib/libvirt/images/${VM_NAME}.qcow2"
export VM_CORES=2
export VM_DISKSIZE=50
export VM_RAMSIZE=4096
export VM_LOCAL_ISO=/tmp/ubuntu.iso

Variable	Purpose
`VM_NAME`	Name of the virtual machine in libvirt
`VM_ISO`	Official Ubuntu 24.04 live server ISO
`VM_OS`	OS variant for optimization
`VM_IMG`	Path to the VM disk image
`VM_CORES`	Number of virtual CPUs
`VM_DISKSIZE`	Disk size in GB
`VM_RAMSIZE`	RAM in MB
`VM_LOCAL_ISO`	Local ISO download path

Download the Ubuntu 24.04 ISO

curl -o "$VM_LOCAL_ISO" -L "$VM_ISO"

Install Ubuntu 24 Using Terminal Installer

Here’s the key part: we instruct virt-install to boot directly from the installer kernel and initrd inside the ISO.

virt-install \
  --virt-type kvm \
  --name "$VM_NAME" \
  --os-variant "$VM_OS" \
  --disk path=${VM_IMG},size=${VM_DISKSIZE},bus=virtio,format=qcow2 \
  --memory "$VM_RAMSIZE" \
  --vcpus "$VM_CORES" \
  --graphics none \
  --console pty,target_type=serial \
  --location ${VM_LOCAL_ISO},kernel=casper/vmlinuz,initrd=casper/initrd \
  --extra-args="console=ttyS0,115200n8 --- console=ttyS0,115200n8"

Ubuntu VM

Creating an Elasticsearch API token for another user

2025-11-01T00:00:00+01:00

Using the superuser elastic API keys to access the Elasticsearch API is not recommended. The API key is often used by remote clients on remote systems. Any attacker who might get access to the token, can compromise the entire Elasticsearch instance.

The solution is to use API keys for unprivileged roles and users. Creating these API keys, however, is not straightforward. Unprivileged users usually don’t have permission to log in and create the API keys for themselves. Run the following request as superuser to create API keys on behalf of unprivileged users, for example, in the /app/dev_tools#/console console.

POST /_security/api_key/grant
{
    "grant_type": "password",
    "username": "elastic",
    "password": "YOUR SUPERUSER PASSWORD",
    "run_as": "UNPRIVILEGED USERNAME",
    "api_key": {
        "name": "NEW NAME FOR THAT API KEY"
    }
}

More details can be found in the API docs.

Dedicated API keys with restricted permissions

Creating tokens for dedicated tasks, for example for Filebeat or Metricbeat clients, can be achieved through another endpoint.

Create an API key for Metricbeat clients with the following request. See the API docs for more details.

POST /_security/api_key
{
  "name": "API KEY NAME", 
  "role_descriptors": {
    "metricbeat_writer": { 
      "cluster": ["monitor", "read_ilm", "read_pipeline"],
      "index": [
        {
          "names": ["metricbeat-*"],
          "privileges": ["view_index_metadata", "create_doc", "auto_configure"]
        }
      ]
    }
  }
}

Create an API key for Filebeat clients with the following request. See the API docs for more details.

POST /_security/api_key
{
  "name": "API KEY NAME",
  "role_descriptors": {
    "filebeat_writer": {
      "cluster": ["monitor", "read_ilm", "read_pipeline"],
      "index": [
        {
          "names": ["filebeat-*"],
          "privileges": ["view_index_metadata", "create_doc", "auto_configure"]
        }
      ]
    }
  }
}

llms.txt adoption

2025-09-03T00:00:00+02:00

Exactly one year ago, Jeremy Howard published a proposal to make the web more accessible to AI and, in particular, to LLMs. How many of the top one million websites adopt this approach?

The proposed standard suggests creating a file at the root of a website, e.g., /llms.txt, intended to be consumed by LLMs and AI tools, loosely taking inspiration from /robots.txt. The /llms.txt serves as an entry point or site map of the website, potentially linking to other pages. As the source code of a webpage is often very verbose and its content is mingled with style sheets, JavaScript, and HTML markup, parsing the source with an LLM might exceed the LLM’s content window or consume too many tokens. Therefore, the idea is to use Markdown for the /llms.txt entry point and to link to Markdown versions of each page.

How many websites adopted this approach?

Let’s measure.

Starting with a dataset of the 10 million highest-ranked domains from Open Page Rank, we can send a GET request to /llms.txt for each of them and see how many web servers respond with an HTTP success code. It turns out, a lot of web servers respond with a 200 status code but actually send a page informing the client that the page doesn’t exist. For example, the top-ranked domain, facebook.com, behaves in that way. A GET request to https://facebook.com/llms.txt returns a 200 status code, but the pages says the content is not available.

A typical feature of these fake-success responses is that the response page is an HTML document. However, sometimes the Content-Type header field is not a reliable discriminator to detect fake-success pages. A good way to distinguish HTML content from Markdown is to compare the number of occurrences of the left angle bracket (<) character to the left square bracket ([). The first is ubiquitous in HTML, while the latter is common in Markdown. I came up with the following somewhat arbitrary rules. Only if a response satisfies all of them, I count it as a valid /llms.txt response.

HTTP response code must be less than 400
Content-Type header must start with text/plain
The web server must accept the connection within 3 seconds
The response must arrive within 10 seconds
The site must support HTTPS
The response content must be longer than 500 chars
There must be more left square brackets than left angle brackets

To speed up the analysis, it limited it to the top one million domains. I ran the analysis on September 2, 2025.

Results

Domains ranked high in the domain ranking might be faster to adopt new technological ideas. To test this, I counted the fraction of domains with /llms.txt among the top n domains. The result is shown in the following chart.

Based on these results, we see that the largest adoption rate at 4 % is among the top 300 domains. The fraction continuously decreases further down the ranking. Looking at the top one million domains, we see that the overall adoption rate drops to around 1.2 %. In total, that corresponds to 12174 domains.

The crawler, the analysis code, and the result dataset are available in a Git repository.

Magic floating-point numbers: NaNs

2025-08-18T00:00:00+02:00

After following Laurie Kirk down a rabbit hole on subnormal numbers in the IEEE 754 float specification, I stumbled upon other interesting properties of floating-point numbers, specifically how NaNs (Not a Number) are represented in binary. After more than 10 years of scientific computing and data science, I thought there was nothing about floats that could surprise me, but oh, was I wrong. Let’s see if I can surprise you. I’ve built the computer-science equivalent of a magic trick to showcase these properties.

The magic trick

The trick works in two stages:

You choose a phrase of your liking. With a special Python function, you can convert it into a numpy array of NaNs. It’s a normal array. It’s normal NaNs. Your phrase is nowhere to be seen.
You send the numpy array to an API endpoint at https://magicfloat.sauerburger.io/unravel. Using advanced magic (knowledge of IEEE 754), I can unravel your secrets by looking at the array of NaNs.

Step one: Enchanting your phrase

import numpy as np

def enchant(phrase: str) -> np.ndarray:
    return np.frombuffer(b"".join([
        bytes([x]) + b"\xff\x80\x7f" for x in phrase.encode("utf-8")
    ]), dtype=np.float32)

If you call that with "Computers are fun!", you get a numpy array of floats with no signs of the phrase. It seems the message is gone.

box = enchant("Computers are fun!")
>>> box
array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
       nan, nan, nan, nan, nan], dtype=float32)

>>> box.shape
(18,)

>>> box.dtype
dtype('float32')

Step two: Open the magic NaN array

I’m providing an API endpoint at https://magicfloat.sauerburger.io/unravel that takes the binary version of the numpy array and responds with your original phrase. The following function does the necessary encoding and request handling.

import requests

def unravel(box: np.ndarray) -> str:
    if box.dtype != np.float32:
        raise ValueError("Magic box must be float32.")
    response = requests.post("https://magicfloat.sauerburger.io/unravel", data=box.tobytes())
    if not response.ok:
        raise RuntimeError("The planets don't seem to align: %s" % response.text)
    return response.text

If we continue the example from above, we get: drum roll

>>> unravel(box)
'Computers are fun!'

How does it work?

Floating-point numbers are represented using three components,

the sign of the numbers,
the exponent used with base 2, and
the fractional part of the number, the mantissa.

In memory, they are arranged as follows. The order might be different depending on the endianness of your platform.

Sign

Biased exponent (8 bits)

Mantissa (23 bits)

A few combinations of bits have a special meaning, such as +inf, -inf, and NaN. When the exponent is all-ones (as shown in the chart), it represents one of the aforementioned three cases.

Sign	Biased exponent	Mantissa	Special meaning
0	all ones: `1111 1111`	all zero	`+inf`
1	all ones: `1111 1111`	all zero	`-inf`
any	all ones: `1111 1111`	at least one bit not zero	`NaN`

We observe that +inf and -inf each have a unique binary representation. However, for NaN, we have 2^24 - 1 possible binary representations. For my little magic trick, I pack one UTF-8 encoded byte in each 32-bit float number. I invite you to discover the details yourself.

LLM basics and its applications

2025-07-16T00:00:00+02:00

Technical innovations often offer numerous applications with tremendous added value, making work easier. However, this also increases the potential for misuse, with the opposite effect—whether deliberately or through improper use. This also applies to advances in the field of Artificial Intelligence (AI). The aim of this article is to shed light on how current text-based AI applications work, so that they can be used meaningfully and appropriately in the field of safety science. The focus here is not to discourage their use, but to encourage an active discussion about sensible applications and their adoption.

Note: The is the written version of my presentation titled Challenge "Artificial Intelligence" in relation to the assessment of safety at XXXIX. International GfS Symposium, Vienna on Mai 21, 2025. The text as translated from German to English using a Large Language Model (LLM) and is not a verbatim transcript of the presentation.

From a user’s perspective, lack of understanding of how AI applications work creates several obstacles. In the following, I will focus on text-based applications powered by Large Language Models (LLMs). Since the launch of ChatGPT in November 2022, it took only a few months before insufficient understanding during use made headlines. As various media reported, a lawyer in New York used ChatGPT to research legal precedents, which he then submitted to the court. It later emerged that most of the cases presented by ChatGPT were either incorrectly cited or completely made up. This behavior, known among experts as “hallucination,” is a characteristic of LLMs. The lawyer claimed to have acted under the assumption that ChatGPT was a search engine. The legal consequences in this case led to the lawyer being fined. Numerous reports of similar cases have emerged over the past two years.

The example above illustrates how important it is to understand how AI applications work. This is not limited to ChatGPT and can also be transferred to other AI tools. Below, I describe how language models function and how modern AI tools are derived from them.

Large Language Models

Large Language Models are neural networks, that is, in a broad sense, nonlinear mathematical functions that compute an output from an input. In the case of language models, both the input and output are text. LLMs are used in almost all generative, text-based AI services. However, they are often hidden behind several layers of application-specific logic. The foundation for today’s models was laid by Google employees in 2017 with the Transformer architecture and the so-called Attention mechanism.

The term “Large” in Large Language Models refers to the number of parameters and the associated memory requirement. There is no clear cutoff to define the term and it is expected that the development of ever larger language models will continue. If the language model is viewed as a mathematical function, the number of parameters becomes comparable: A first-degree polynomial, i.e. a function that describes a straight line in a plane, has two parameters; a second-degree polynomial (parabola) has three. In general, GPT-1 is considered the first LLM, which is described by 117 million parameters. Today’s models have up to 700 billion parameters, requiring specialized hardware for their application. Further examples of large language models include OpenAI’s GPT-4.1 or o3, as well as openly available models such as BLOOM, Llama 3, and Mixtral.

Functionality

LLMs do not work directly with words or characters, but with “tokens”—essentially the alphabet of the language model. A token reflects a semantic unit of a word and is the smallest element the model understands. In English, a token is often equated to about ¾ of a word, so on average about 1⅓ tokens are required to form a word. For a language model to process text, it is first broken down into a sequence of tokens and each token is identified by an integer. The original text is thus translated into a chain of numbers. Typically, the vocabulary of modern language models contains about 20,000 to 200,000 different tokens.

Large language models for generative applications are mostly trained to predict the next token based on a given sequence of tokens—so-called Next Token Prediction. Put simply, the goal of training is to optimize the vast number of parameters in the language model so that it can predict the next token for texts from the training corpus as accurately as possible. To train such a large number of parameters, a correspondingly large dataset of texts is required. For high precision, the language model must be capable of understanding context both within sentences and across entire texts. Earlier approaches in computational linguistics, such as Markov chains or neural networks with Long Short-Term Memory (LSTM), do not achieve comparable results.

Sticking with the view of language models as mathematical functions, so far the model appears to compute the next token from a sequence of input tokens. By repeatedly applying the function, always appending the predicted token to the input sequence, it is possible to continue or complete a started text. This is illustrated in the following figure. To enhance readability, input and output texts are shown as plain text rather than tokenized. The model completes the started sentence. Although this appears to work well at first glance, the phenomenon of hallucination also appears here, as the described “FINTURBO Cup” does not exist.

Mental Model for the Model

According to a 2022 survey, computational linguists were divided on whether LLMs truly “understand” natural language in a non-trivial sense, or merely remix and parrot back training texts. It is unclear how language models solve linguistic tasks, and they are often described as black boxes.

The author believes this question is not productive, as it enters deeply philosophical territory. For instance, to answer it, we would first need to define what is meant by “understanding” and to what extent a machine can actually understand anything.

From a practical perspective, it makes sense to use the following mental model for LLMs: Since LLMs are above all trained to predict the next word or token, it is reasonable to say that LLMs merely imitate human language and conversation. It shouldn’t be surprising that such imitation can still be extremely useful for solving tasks. Throughout this article, I will refer back to this perspective to explain the behavior of LLMs in specific examples.

Techniques

The following sections highlight various techniques, all of which are applied in one form or another in today’s AI tools.

Paradigm Shift: Prompt Engineering

Programming has traditionally meant that a computer executes the instructions in program code exactly as written. This can be surprising when, for example, an “obviously” correct algorithm is carried out by the machine differently than a human would expect. Translating abstract ideas into concrete, specific computer instructions is a core part of programming.

Prompt Engineering marks a radical departure from this principle. When language models are used, instructions are no longer executed with mathematical precision. Prompt Engineering refers to the practice of writing instructions—called prompts—to a language model so that it completes a task as reliably and accurately as possible. A prompt is executed by the model in the context of simulated conversation. Small, seemingly insignificant changes to the prompt’s wording can lead to radically different answers. For instance, if a language model’s output must adhere to a specific format to work with traditional software, it is not uncommon to remind the model several times within the prompt about the desired output format, as shown in the following figure.

LLMs tend to invent or guess answers when they do not know the correct response—this is the already described phenomenon of hallucination. Providers of AI services attempt to avoid or reduce hallucination in their products. Because of this tendency, pure language models are not suitable as factual knowledge bases or search engines. Ignorance of this property led to the initial example with the lawyer.

As radical and paradoxical as the departure from mathematical precision in programming may seem, this paradigm shift dramatically increases the range of possible applications. Writing a short textual instruction to an LLM can solve problems that could not be addressed with conventional programming, or only through great effort using traditional computational linguistics techniques. Applications can more quickly adapt to changing requirements simply by changing prompts, whereas retraining a classic computational linguistics model takes significant time.

In-Context Learning

Language models have a limited knowledge base, known as the knowledge cut-off. During training, a large corpus of texts is used. The model cannot know anything about events not included in its training set, for example because they occurred after training was completed. Language models generally cannot access all information from the training dataset as a knowledge base would. The purpose of the dataset is to teach the model language and conversation—not to memorize facts and events.

Since training LLMs is time- and resource-intensive, there are other ways to convey new information to them. A fundamental method is so-called in-context learning, in which, at the start of a conversation with the language model—i.e. after training is finished—the model is given all necessary information for the task as part of the input text. This can include background knowledge required for a task, or examples that demonstrate how the task should be solved (“few-shot learning”). The following figure shows a conversation in which the language model is taught, using examples, how to rephrase sentences.

External Knowledge and Logical Reasoning

The concept of in-context learning can be extended and combined with classic knowledge bases. Information required for answering a question or carrying out a task is added to the conversational context by extracting it from a knowledge base. This technique is called Retrieval Augmented Generation (RAG). RAG applications differ greatly depending on the data source. A RAG application can consult a single document, an encyclopedia like Wikipedia, or all freely available internet content.

Developing the retrieval aspect is essential for a successful RAG system. If the extracted content does not include the information needed to answer the question, the language model cannot provide a meaningful response. RAG is often seen as another method to reduce LLM hallucinations. The following illustrates the workings of a RAG system with an example.

The analogy—that language models merely imitate conversation—points to another limitation that was humorously noted online in the early days of ChatGPT: the lack of logical reasoning ability.

THe following figure illustrates that the language model did not reach the correct logical or mathematical conclusion. The example uses in-context learning to demonstrate what the model is being asked to do. It remains unclear how the model arrived at the answer “27”; the correct answer is 9.

A common technique to guide language models toward logical reasoning is called Chain-of-Thought Prompting (COT prompting), in which the model is encouraged to document and explain intermediate steps. Since of the way the model works, each subsequent step is generated only after preceding intermediates have been “put on paper” and made part of the input. THe following figures shows the same task, this time solved correctly with COT prompting.

Agents and Fine-Tuning

Innovations in prompt engineering and related techniques allow a language model to be used for a wide range of applications. However, its capabilities are limited to pure text (or audio/image) output. Interaction with external systems requires so-called agents, which can enable complex computations and allow language models to interact with external systems.

Through prompt engineering, a language model is told, in its instruction, what external tools are available for solving a task. This could include a calculator, an external API (to manage emails, appointments, or contacts), a RAG-based search, a free web search, or a coding environment.

Users can assign a task to the language model. The model is instructed to select, from the available tools, the one required to solve the problem. The model communicates the selection and use of tools via a special answer format. Complex systems allow for the use of multiple tools to solve the task step by step.

A language model with internet search and route planner abilities could, for example, handle the request “How long does it take to drive from Vienna to the host venue of the Eurovision Song Contest 2025?” by first using a web search to find where Eurovision will be held in 2025, then using the route planner to calculate the travel time from Vienna to Basel. The answer is about eight hours.

The potential of agent-based applications is virtually limitless in our connected digital world. Today’s language models are often trained on both natural and programming languages, allowing them to generate functional code. In combination with a programming environment, language models can write complex algorithms needed to solve a task and then execute them. The limits of such systems in the future are currently unforeseeable.

All the techniques discussed so far use prompt engineering to influence the work of the language model. To conclude, there is another technique that does not rely on prompt engineering: Through fine-tuning, a pre-trained language model can be customized so that it automatically follows a fixed, predefined instruction. In a way, the model loses its ability to react generically to all prompts. On the other hand, the model doesn’t have to be told its task at the start of every conversation. Fine-tuning is suitable when a language model will be used for a large number of identical tasks. Depending on the fine-tuning method, all the parameters of the model may be adjusted. However, this is often too computationally expensive. Alternatives like PEFT enable fine-tuning by adding or adjusting only a small number of new parameters.

Final Thoughts

Meaningful use of AI in safety sciences requires understanding both the underlying technology and its limitations. By presenting some essential techniques used in today’s AI tools, this article aims to make a contribution here.

The landscape of commercial and open-source language models, and their abilities and methods, is changing rapidly. It is difficult to make predictions about the future of AI. But AI will certainly change the way many industries work, and companies that fail to use AI tools will have a hard time competing. It is therefore all the more important to seize the opportunities that AI provides. To make this easier, this article has highlighted some technical boundaries, to help avoid mistakes in application.

Restore a collection from a Qdrant snapshot stored in S3

2025-02-26T00:00:00+01:00

The lightning-fast vector database Qdrant has supported creating snapshots of its collection on S3 since version v1.10. That’s convenient and simplifies storing snapshots as backups on multiple machines. However, as of 2025, there is no way to restore a collection from a snapshot on S3. This article describes a walkaround using pre-signed URLs in S3.

Creating the snapshot

A Qdrant instance can be configured to use S3 as its storage for snapshots.

storage:
  snapshots_config:
    snapshots_storage: s3
    s3_config:
      bucket: your_bucket_here
      region: your_bucket_region_here
      access_key: your_access_key_here
      secret_key: your_secret_key_here
      endpoint_url: your_url_here

The next snapshot request will be written to the S3 bucket.

from qdrant_client import QdrantClient

client = QdrantClient(url="http://localhost:6333")
snapshot_info = client.create_snapshot(collection_name="my-collection")

Creating a pre-signed URL

From the snapshot_info object, we can figure out the name of the snapshot within the bucket. Alternatively, inspecting the bucket directly works as well. Let’s assume, the snapshot is called

snapshots/my-collection/my-collection_123456789.snapshot

Since Qdrant is not able to read and restore a snapshot from S3 directly, we can use S3’s pre-signed URLs. A pre-signed URL contains a signature that authorizes everyone with the URL to access a specific object in the bucket—assuming that the API and the bucket is not configured to be entirely private.

AWS’s Python client boto3 is an easy way to create pre-signed URLs.

import boto3


session = boto3.session.Session()
s3 = session.client(
        service_name='s3',
        aws_access_key_id='...',
        aws_secret_access_key='...',
        endpoint_url='...',
)

url = s3.generate_presigned_url(
        'get_object',
        Params={
            'Bucket': '...',
            'Key': 'snapshots/my-collection/my-collection_123456789.snapshot',
        },
        ExpiresIn=3600 * 24 * 7,  # Expires in 7 days
)

Creating pre-signed URLs doesn’t require internet access. The key id and access key pair is sufficient to create the signature offline. The S3 server validates the signature and evaluates permissions once the resource is requested.

The HTTP URL points to the snapshot file. This could look like

>>> url
'https://s3.sauerburger.com/' \
    'qdrant-backup/snapshots/my-collection/my-collection_123456789.snapshot?' \
    'X-Amz-Algorithm=AWS4-HMAC-SHA256&' \
    'X-Amz-Credential=a4tw715etr74zq2dhxdp70s1%2F20250212%2Feu-central-1%2Fs3%2Faws4_request&' \
    'X-Amz-Date=20250212T194041Z&' \
    'X-Amz-Expires=604800&' \
    'X-Amz-SignedHeaders=host&' \
    'X-Amz-Signature=23ea0fcfa91bb5e3ebea847e79cc69f08224e58275890097e28bed9e1de018df'

Restoring a collection

The final step is to instruct Qdrant to restore a collection from a snapshot via HTTP.

from qdrant_client import QdrantClient

client = QdrantClient(url="http://localhost:6333")
success = qdrant.recover_snapshot(
    "my-collection",
    url,
    wait=True,
    priority='snapshot'
)

Done.

Summary

Qdrant → S3 → Pre-signed HTTP URL → Qdrant

Network analysis with Scapy and Polars

2025-01-29T00:00:00+01:00

Sometimes, debugging state-of-the-art AI applications in an on-premise Kubernetes cluster requires capturing network packets and performing complex statistical traffic exploration and analysis. Traffic is easily captured with

sudo tcpdump -i any -s 65535 -w /tmp/capture.pcap

and decoded with Wireshark. However, complex analyses require other tools. Let’s open the data scientists’ toolbox: Polars.

Loading

The basic idea is to use Scapy to read the capture file, decode the packets and various protocols, and organize the data in a Polars dataframe. In this example, let’s extract the source and destination IP address, the packet length, and the query domain name from DNS packets.

import polars as pl
import matplotlib.pyplot as plt
import scapy.all as sa
from scapy.all import PcapReader
import seaborn as sns
from tqdm import tqdm

with PcapReader("capture.pcap") as reader:
    df = pl.DataFrame(
        (
            {
                "IP:src":     packet["IP"].src                    if "IP"  in packet else None,
                "IP:dst":     packet["IP"].dst                    if "IP"  in packet else None,
                "IP:len":     packet["IP"].len                    if "IP"  in packet else None,
                "DNS:qcode":  packet.sprintf("%DNS.opcode%")      if "DNS" in packet else None,
                "DNS:qnames": [x.qname for x in packet["DNS"].qd] if "DNS" in packet else [],
            }
            for packet in tqdm(reader)
        ),
        schema=pl.Schema({
            "IP:src": pl.String(),
            "IP:dst": pl.String(),
            "IP:len": pl.Int32(),
            "DNS:qcode": pl.String(),
            "DNS:qnames": pl.List(pl.String()),
        })
    )

# Derive additional columns
df = df.with_columns(
    internal=pl.col("IP:src").str.starts_with("10.") & pl.col("IP:dst").str.starts_with("10.")
)

Additionally, we can derive additional columns, such as whether the packet is internal or external, based on the IP addresses. A more robust analysis could include the packet’s IP addresses as 32-bit integers and applying bitwise operations to determine membership in a network subnet.

The resulting, redacted dataframe looks something like:

IP:src	IP:dst	IP:len	DNS:opcode	DNS:qnames	internal
“10.x.x.x”	“10.x.x.x”	128	null	[]	true
“162.55.242.49”	“91.59.x.x”	188	null	[]	false
“10.x.x.x”	“10.x.x.x”	93	null	[]	true
“10.x.x.x”	“10.x.x.x”	93	null	[]	true
“10.x.x.x”	“10.x.x.x”	843	null	[]	true
“10.x.x.x”	“10.x.x.x”	139	“QUERY”	[“ns-2.sit-servers.net.”]	true

Packet length analysis

So far so good. Assume we want to investigate elevated retransmission rates. We might want to look at the distribution of packet lengths, for internal and external traffic. With the current setup, we can hand the dataframe to seaborn for visualization.

import seaborn as sns

sns.histplot(df, x="IP:len", bins=40, hue='internal', element="step")
plt.yscale("log")
plt.xlabel("Packet size / Bytes")

DNS server analysis

Next, we might want to investigate the DNS queries. Let’s look at the frequency of query names. Since we captured traffic on all interfaces, we want to filter out queries for internal servers. That’s easily done with Polars. Furthermore, since we don’t specify the direction of the query, incoming or outgoing, we capture both: incoming DNS queries to the authoritative server where tcpdump was running, as well as, name lookups originating from the server.

dns_stats = (
    df
    .explode("DNS:qnames")["DNS:qnames"]
    .drop_nulls()
    .value_counts()
    .filter(
        pl.col("DNS:qnames").str.ends_with("in-addr.arpa.").not_() &
        pl.col("DNS:qnames").str.ends_with("local.").not_()
    )
)
dns_stats.sort("count", descending=True)

DNS:qnames	count
“gitlab.sauerburger.com.”	91
(redacted)	55
“ns-1.sit-servers.net.”	30
“ns-2.sit-servers.net.”	30
“frank.sauerburger.io.”	12
“fjell.ai.”	8
“neodns.io.”	8
“sauerburger.io.”	8
“debugci.dev.”	6
“www.fjellai.cloud.”	6
“sAUeRbuRgEr.DeV.”	6
“ds.sit-servers.net.”	6
“nEodns.teCH.”	6
“Ns-1.sIT-servErs.neT.”	6
“NS-2.SiT-SErVerS.NeT.”	6
“ns-2.sit-SeRveRS.nET.”	6
“.uhepp.org.”	6
“net.stratus.sit-servers.net.”	6
“NS-1.sIT-SERVErs.NEt.”	2
“neODNS.TecH.”	2
“nS-1.siT-SerVERS.NEt.”	2

If you’re wondering why some of the DNS entries have random captialization, that’s just Google focussing on its core business.