slash dev slash null

stuff about puters

 Corfu: Global Sequencer

This is the third post in a series about Corfu. In the last post we outlined how to linearizable consistency without a global buffer. This introduced a write contention challenge. In this post we will get into Corfu's simple fix for this issue. Corfu uses a global ticket service which acts as a sequencer:

Read the rest of this entry »

Corfu: Linearizable Consistency

This the second post in a series about Corfu. To quote the paper

The setting for CORFU is a data center with a large number of application servers (which we call clients) and a cluster of storage units. Our goal is to provide applications running on the clients with a shared log abstraction implemented over the storage cluster.

Why is a shared log abstraction exposed to clients useful? This post will discuss how a shared log can be used to achieve linearizable consistency. To quote the paper:

The semantics provided for read/write operations is linearizability [Herlihy and Wing 1990], which for a log means that once a log entry is fully written by a client or has been read, any future read will see the same content (unless it is reclaimed via a [delete] operation).

We will then look at a simple adaption that we will later see can be used to scale the system which introduces an additional challenge to solve.

Read the rest of this entry »

Corfu: Scaling log replication

Corfu: A distributed shared log is a paper running to 25 pages brought to my attention by The Morning Paper. What I really like about Corfu is how it borrows bits and pieces of established techniques and puts them together to get something which is a real breakthrough. Corfu achieves linearizable consistency of a fault tolerant distributed log using very simple write-once techniques. The real eye open is that it implements a distributed log interface without an IO bottleneck giving extreme performance. To quote the papers conclusion:

Despite almost forty years of research into replicated storage schemes, the only approach so far to scale up capacity and throughput has been to shard data and trade consistency for performance. In this article, we presented the CORFU system, which breaks this seeming tradeoff by organizing a cluster of drives as a single, shared log. CORFU offers single-copy semantics at cluster-scale speeds, providing a scalable source of atomicity and durability for distributed systems. CORFU’s novel client-centric design eliminates any single I/O bottleneck between numerous clients and the cluster, allowing data to stream to and from the cluster in parallel.

 

This series of posts will build up a simple fictional example that demonstrates the key techniques. Then I will glue it all together to describe how Corfu itself works and demystify the role Paxos plays.

Read the rest of this entry »

C# ASP.NET with SQLServer on Docker and Linux is now “A Thing™”

Those of you who have been following along will know that Scala on Linux is my preferred ecosystem. This past few weeks in the office I have been tinkering with the opensource C# ASP.NET ecosystem. What I came across shocked me to the dotnet core. I have posted some evidence of my findings up on GitHubRead the rest of this entry »

TRex UPaxos Experimental Code

There is now a sketch (hack!) demoing the happy path flow of a UPaxos reconfiguration written up on the TRex wiki.

Paxos Voting Weights

The last UPaxos post took a run through the reconfiguration stall avoiding aspects of the UPaxos paper. In this post, we will take a quick look at how the paper describes hot swapping of nodes using voting weights. Read the rest of this entry »

Cassandra For Shared Media Libraries

A good friend of mine is working on a project which hosts media libraries in his cloud service. At the end of 2015, I integrated Cassandra into a big financial services platform. Cassandra is a great fit for my friend’s service. In this post, I will outline an appropriate Cassandra data model and along the way outline some of the killer features of Cassandra. Read the rest of this entry »

The End Of Agile Estimation: Lies, Damn Lies, And Burndown Charts

In my last couple of posts, I gave an example of a successful agile team that didn’t do estimates, then made the case that estimation is aligned to “wagile” ways of working. This post is going to make the case that the approach is fundamentally flawed when viewed as a measurement activity.

Read the rest of this entry »

The End Of Agile Estimation: Predictability Isn’t A “Good Thing™”

In the last post I gave an overview of large scale build which used Kanban and that shunned estimates. In this post I will get into why Agile estimation is not aligned to agile values. It is simply a hangover from waterfall days and is a “wagile” concept.  Read the rest of this entry »

The End Of Agile Estimation: Flying Without Estimates

In the past twelve months I have had the privilege of working across more than a half-dozen teams building digital services. It was interesting that only one of them was attempting to do estimates and it wasn’t clear to anyone, let alone them, exactly why they were attempting it. In this post I will sketch out how a successful large scale team abandoned estimations entirely.  Read the rest of this entry »