slash dev slash null

stuff about puters

Pre-voting in distributed consensus

Another top notch Paxos post from the inventor of UPaxos covers leader election in Paxos. The outlined approach is similar to that used in TRex which is based on a sloppy timeout mechanism. This post will get into why this is a must read for consensus fans. Read the rest of this entry »

Observability in Paxos clusters

David Turner the inventor of UPaxos has posted a great blog post about monitoring Paxos clusters. It’s a must read for consensus aficionados.

Rust and Embedded Databases for Paxos 

So far on my spike into Rust we have been on a roll. Next on the bucket list is an embedded disk backed B-tree database to act as the Paxos journal for TRex. This is where I have hit my first bump in the road. Read the rest of this entry »

Rust Extension Methods

Seven years ago I was working on a system that had a fat client desktop app written in C# and a Java backend. I signed up for a week-long C# training programme. I was fascinated about the bits of C# that had evolved away from Microsoft J++. A lot of the differences that still stuck in my mind were all about boilerplate removal. One of those was C# Extension Methods. I have posted some sample code of Rust Extention Methods over on GitHub.    Read the rest of this entry »

Rust Pickling 

The first thing item to research on my Rust spike is picking. A quick survey of the landscape indicates that serde appears to be the current defacto standard framework. Pickling was an area that I chose to hand code in the TRex Scala implementation. Why? Because Scala’s pickling engine crashed my JVM and wasn’t yet the stage of a stable disk format. I want to have no external dependencies for the inner Paxos library and as few as possible elsewhere. As Scala’s official pickling engine wasn’t ready for prime time, I was on my own. I had some fun writing my own ByteChain based pickling where I wrote a compact binary wire encoding for unsigned integers. With Rust does serde put me in a better place? Read the rest of this entry »

Why Rust for TRex?

My programming career started out with C and C++ last century. I then switched to Java for a long while then finally moved onwards and upwards into Scala. Along the way I have put JavaScript into production and written a JavaScript authentication library others use. There was also a bit of PHP on the side which I don’t draw attention to. Why not return to C++ for a systems programming language port of Trex? Or try D? Why Rust?  Read the rest of this entry »

Rust FTW?

It has been on the back of my mind for a while that Scala might not be too many folks first choice language to implement a Paxos library. So I have been considering the options for porting TRex onto a systems programming language.

A couple of weeks ago I started reading The Rust Programming Language. Before I knew it I have been sucked in and find myself mildly, and occasionally widely, enthusiastic about Rust. There are still a lot of little steps I need to test out, and lessons I need to learn before I make the decision to do a full port. I thought to blog about those steps and lessons as I go long. For my next post, I will explain why I started out looking at Rust rather than another language.

Corfu: The Protocol

This is the eighth and last in a series of posts about Corfu a strongly consistent and scalable distributed log. To quote the paper:

At its base, CORFU derives its success by scaling Paxos to data center settings and providing a durable engine for totally ordering client requests at tens of Gigabits per second throughput.

It does this through a number of existing techniques:

Our design borrows bits and pieces from various existing methods, yet we invented a new solution in order to scale SMR [state machine replication] to larger cluster scales and deliver aggregate cluster throughput to hundreds of clients.

What I really like about Corfu is that it innovates by bringing existing techniques such as Paxos together into something which is a real breakthrough. The protocol itself is very simple. The established techniques it uses to implement the protocol are very simple. Yet when assembled into the solution they become extremely powerful. This post will put the pieces together and discuss the Corfu protocol.

Read the rest of this entry »

Corfu: Paxos Configurator

This the seventh post in a series about the techniques used in Corfu. The techniques described this far are the basic mechanics. Corfu uses these to build a linearizable log with one final ingredient: Paxos. To quote the paper:

At its base, CORFU derives its success by scaling Paxos to data center settings and providing a durable engine for totally ordering client requests at tens of Gigabits per second throughput.

It does this by using Paxos to control the cluster configuration whilst using the simpler techniques outlined in this series of posts to scale the main transactional load.  Read the rest of this entry »

Corfu: Safety Techniques

This the sixth post in a series about Corfu. It will cover some of the basic safety techniques which I call “Write Once”, “Write-Ordering” and “Collaborating Threads”. What is remarkable about Corfu is how it uses such well-established techniques to build something extremely scalable. Read the rest of this entry »