This can be used to create hilly terrain in a video game, or more generally, it’s used in computer graphics for procedurally generating textures or making bumpy surfaces. The original version of Minecraft used a 2-d noise function to generate heights for terrain as a function of (x,y) coordinates, and later versions used a 3-d noise function to generate terrain density as a function of (x,y,z) coordinates to produce overhangs and caves, as Notch describes here.

There’s many different techniques for making noise. The simplest is called “Value Noise” and it works by generating a set of random points, and then interpolating between them. Most implementations store the random points in a fixed-width array, and points past the end of the array wrap around. Value noise looks like this:

class ValueNoise: NUM_VALUES = 256 SPACING = 10 def __init__(self, random=random): self.values = [random() for _ in range(ValueNoise.NUM_VALUES)] def evaluate(self, x): # Look up the nearest two values x1 = floor(x/ValueNoise.SPACING) x2 = ceil(x/ValueNoise.SPACING) n1 = self.values[x1 % ValueNoise.NUM_VALUES] n2 = self.values[x2 % ValueNoise.NUM_VALUES] # Smoothly interpolate between them using cosine interpolation k = .5 - .5*cos((x/ValueNoise.SPACING % 1)*pi) return ((1-k)*n1 + k*n2)

More commonly, Perlin Noise and Simplex Noise are used. They are slightly more complicated algorithms that rely on similar principles: generate a large amount of random values, then look up and combine those values to get the value at a particular point. However, with Perlin and Simplex noise, the values are used as gradients, rather than heights, which produces slightly better-looking noise.

For many applications, it’s nice to have some extra detail. For example, if you’re using a noise function to make mountains, it’s good to have both “general trend” noise like “go upward for 50 miles, then go downward again”, but it’s also useful to have “local variation” noise like “there’s a bump on the road here”. Commonly, this is achieved by using multiple layers of noise. You start with high-amplitude, slow-changing noise (mountains) and then add smaller, quicker-changing noise (hills), and so on until you get to the smallest level of detail that you care about (bumps in the road). You can use the same noise function for each layer, if you scrunch it up by scaling the input and output of the function. The layers are commonly called “octaves”, and it looks like this:

class OctaveNoise: def __init__(self, num_octaves, random=random): self.noise = ValueNoise(random=random) self.num_octaves = num_octaves def evaluate(self, x): noise = 0 for i in range(self.num_octaves): size = 0.5**i noise += size * self.noise(x/size) return noise/2

Unfortunately, most of the existing noise functions require pre-generating large amounts of random values that need to be interpolated between (commonly 256). If you keep sampling the noise function far enough in any one direction, the noise function will start repeating itself. Also, most noise functions don’t make any guarantees about the distribution of values. For example, Value Noise picks random points and interpolates between them, but most of the interpolated points are closer to 0.5 than to either 0 or 1, so the distribution of produced values is skewed towards the middle, and it’s quite rare to see the noise function get very high or very low, but quite common to see it pass through 0.5. Perlin Noise and Simplex Noise also suffer from these issues, though they do tend to produce better-looking results than Value Noise. For example, here’s what 1D Simplex Noise looks like:

Notice that the values produced by the Simplex Noise tend to be very heavily biased towards 0.5, and it’s very uncommon to see values close to 0 or 1. (The implementation here is based on the SimplexNoise1234 implementation used in the LÖVE game engine.)

As an alternative to address some of these issues, I present a new noise function, which I call “Hill Noise”. It’s inspired by Fourier Decomposition: any periodic function can be broken down into a sum of sine waves of different amplitudes, wavelengths, and offsets. However, if you generalize the idea, you can create an arbitrarily complex and wobbly function by summing together a bunch of sine waves of different amplitudes, offsets, and wavelengths. If you sum together a bunch of sine waves, then the pattern will only repeat itself after the least common multiple of the periods of the sine waves. For example, the function y = sin(x/2) + sin(x/3) has a period of 6*(2*pi). But the function y = sin(x*100/201) + sin(x*100/301) has a period of 605.01*(2*pi), which is *much* larger because the two sine waves are slightly misaligned and take a long time to resynchronize. For two sine waves with randomly chosen periods, the resulting function typically has an incredibly large period. Even if the periods aren’t randomly chosen, the very high period functions can be produced easily. For example: sin(x*1) + sin(x*.91) + sin(x*.91^2) has a period of 10,000 * (2*pi).

class HillNoise: def __init__(self, num_sines, random=random): self.wavelengths = [random() for _ in range(num_sines)] self.amplitudes = [1 for _ in range(num_sines)] def evaluate(self, x): noise = 0 for amplitude,wavelength in zip(self.amplitudes, self.wavelengths): noise += amplitude*sin(x/wavelength) return .5 + .5*noise/sum(self.amplitudes)

The result of this is not terribly satisfying, because some of the sine waves are undesirably steep. If we’re trying to mimic the bumpiness of real life things, you typically don’t see bumps in the road that are as tall as a mountain, but as wide as a pebble. Generally, the shorter the length of a bump, the shorter the height of a bump. So, in order to emulate this, instead of sin(x/k), let’s use k*sin(x/k), which makes the amplitude directly proportional to the wavelength. If you add up an infinite number of these sine waves with inverse power of two amplitudes, you get a Weierstrass Function, which is starting to look pretty noisy.

class HillNoise: def __init__(self, num_sines, random=random): self.sizes = [random() for _ in range(num_sines)] def evaluate(self, x): noise = 0 for size in self.sizes: noise += size*sin(x/size) return .5 + .5*noise/sum(self.sizes)

It’s a bit unfortunate that this noise function always evaluates to exactly 0.5 at x=0, so let’s introduce some random offsets. (This will be even more useful later.)

class HillNoise: def __init__(self, num_sines, random=random): self.sizes = [random() for _ in range(num_sines)] self.offsets = [random()*2*pi for _ in range(num_sines)] def evaluate(self, x): noise = 0 for size,offset in zip(self.sizes, self.offsets): noise += size*sin(x/size + offset) return .5 + .5*noise/sum(self.size)

Now, this function is already pretty useful, and it comes with built-in octaves with different levels of detail. But, ideally it would be nice if you could ask for different distributions of bumpiness. Maybe you don’t care about any noise whose amplitude is less than 0.1, or maybe you want a lot of small noise and a lot of big noise, but not much in between. So, let’s modify the function to take in an arbitrary set of wavelengths:

class HillNoise: def __init__(self, sizes, random=random): self.sizes = sizes self.offsets = [random()*2*pi for _ in range(len(sizes))] def evaluate(self, x): noise = 0 for size,offset in zip(self.sizes, self.offsets): noise += size*sin(x/size + offset) return .5 + .5*noise/sum(self.size)

Because we added the random offsets earlier, we can now create two different noise functions with the same wavelengths, but which produce totally different noise! Nice!

So far, this is looking pretty good, but when we throw in a *whole lot* of sine waves of similar amplitude, they kind of tend to average out and we get a very bland noise function. According to the Central Limit Theorem, randomly sampling points on this noise function will have an approximately normal distribution, and the standard deviation will get smaller and smaller when more sine waves are added. In order to counteract this, we need to use the cumulative distribution function of the normal distribution. Unfortunately, it’s rather tricky to compute the cumulative distribution function, but luckily we don’t need to be very exact, and Wikipedia conveniently has an entire section on numerical approximations of the normal CDF. Using a little bit of math black magic, it’s possible to calculate the standard deviation of the sum of the sine waves, and plug it into the CDF approximation.

class HillNoise: def __init__(self, sizes, random=random): self.sizes = sizes self.offsets = [random()*2*pi for _ in range(len(sizes))] self.sigma = sqrt(sum((a/2)**2 for a in self.amplitudes)) def evaluate(self, x): noise = 0 for size,offset in zip(self.sizes, self.offsets): noise += size*sin(x/size + offset) # Approximate normal CDF: noise /= sigma return (0.5*(-1 if noise < 0 else 1) *sqrt(1 - exp(-2/pi * noise*noise)) + 0.5)

The end result of this is noise that is still quite noisy, but now it’s (approximately) evenly distributed between 0 and 1:

With this evenly distributed noise function, it’s fairly easy to create noise with any precise distribution that you want using inverse transform sampling. For example, here’s a Weibull distribution (with adjustable parameters alpha and beta):

class WeibullHillNoise: def __init__(self, sizes, alpha, beta, random=random): self.hill_noise = HillNoise(sizes, random) self.alpha, self.beta = alpha, beta def evaluate(self, x): u = 1.0 - self.hill_noise.evaluate(x) return self.alpha * (-math.log(u)) ** (1.0/self.beta)

That’s quite powerful! And it’s not something that can be easily achieved with just any old noise function.

Now, this is a lovely noise function, but it only works for 1 dimension of input. Let’s expand it to work in 2 dimensions! Instead of summing up sin(x) values, let’s sum up (sin(x)+sin(y))/2 values.

class HillNoise2D: def __init__(self, sizes, random=random): self.sizes = sizes self.offsets = [random()*2*pi for _ in 2*range(len(sizes))] self.sigma = sqrt(sum((a/2)**2 for a in self.amplitudes)) def evaluate(self, x, y): noise = 0 for i,size in enumerate(self.sizes): noise += size/2*(sin(x/size + self.offsets[2*i]) + sin(y/size + self.offsets[2*i+1])) # Approximate normal CDF: noise /= 2*sigma return (0.5*(-1 if noise < 0 else 1) *sqrt(1 - exp(-2/pi * noise*noise)) + 0.5)

These will be axis-aligned, which is problematic because it will make the noise function much less random-looking, so for each 2D sine wave, let’s rotate its coordinates by a random angle.

class HillNoise2D: def __init__(self, sizes, random=random): self.sizes = sizes self.offsets = [random()*2*pi for _ in range(len(sizes))] self.rotations = [random()*2*pi for _ in range(len(sizes))] self.sigma = sqrt(sum((a/2)**2 for a in self.amplitudes)) def evaluate(self, x, y): noise = 0 for i,size in enumerate(self.sizes): # Rotate coordinates rotation = self.rotations[i] u = x*cos(rotation) - y*sin(rotation) v = -x*sin(rotation) - y*cos(rotation) noise += size/2*(sin(u/size + self.offsets[2*i]) + sin(v/size + self.offsets[2*i+1])) # Approximate normal CDF: noise /= 2*sigma return (0.5*(-1 if noise < 0 else 1) *sqrt(1 - exp(-2/pi * noise*noise)) + 0.5)

That looks pretty good, but sometimes the randomly generated rotations are nearly aligned for similar-sized sine waves, and it causes some unfortunate aliasing. Really, we don’t need the rotations to be random, but we want them to be fairly evenly distributed, and similarly sized sine waves should have not-parallel, not-perpendicular rotations. There’s a fairly clever trick for doing almost exactly this task, that I learned about in this blog post about generating random colors programmatically. Let’s use that technique instead of purely random rotations.

GOLDEN_RATIO = (sqrt(5)+1)/2 class HillNoise2D: def __init__(self, sizes, random=random): self.sizes = sizes self.offsets = [random()*2*pi for _ in 2*range(len(sizes))] self.sigma = sqrt(sum((a/2)**2 for a in self.amplitudes)) def evaluate(self, x, y): noise = 0 for i,size in enumerate(self.sizes): # Rotate coordinates rotation = (i*GOLDEN_RATIO % 1)*2*pi u = x*cos(rotation) - y*sin(rotation) v = -x*sin(rotation) - y*cos(rotation) noise += size/2*(sin(u/size + self.offsets[2*i]) + sin(v/size + self.offsets[2*i+1])) # Approximate normal CDF: noise /= 2*sigma return (0.5*(-1 if noise < 0 else 1) *sqrt(1 - exp(-2/pi * noise*noise)) + 0.5)

Here’s what the finished product looks like:

Great! Now what about 3D noise? Well… it gets a fair bit more complicated. For one thing, it’s fairly hard to visualize 3D noise, because it has a different value at every point in 3D space. It’s a lot harder to notice if there are any artifacts. So instead, I’m going to visualize it as 2D noise and use time as the 3rd dimension to produce a noise function that wobbles randomly over time. The trick we used for evenly spacing the rotations last time isn’t going to work anymore, because now we need random polar coordinate rotations. With a lot of experimentation, and thanks to the excellent paper Golden Ratio Sequences For Low-Discrepancy Sampling, I managed to come up with a way to generate evenly spaced polar rotations that satisfy the same requirements we had for 2D noise. The end result is starting to get a bit hairy, but it looks like this:

GOLDEN_RATIO = (sqrt(5)+1)/2 class HillNoise3D: def __init__(self, sizes, random=random): self.sizes = sizes self.offsets = [random()*2*pi for _ in range(len(sizes))] self.sigma = sqrt(sum((a/2)**2 for a in self.amplitudes)) def evaluate(self, x, y): fib_num = floor(log((resolution-1)*sqrt(5) + .5)/log(GOLDEN_RATIO)) dec = floor(.5 + (GOLDEN_RATIO**fib_num)/sqrt(5)) inc = floor(.5 + dec/GOLDEN_RATIO) noise = 0 j = 0 for i in range(len(self.sizes)): if j >= dec: j -= dec else: j += inc if j >= len(self.sizes): j -= dec # Convert golden ratio sequence into polar coordinate unit vector phi = ((i*GOLDEN_RATIO) % 1) * 2*pi theta = acos(-1+2*((j*GOLDEN_RATIO) % 1)) # Make an orthonormal basis, where n1 is from polar phi/theta, # n2 is roated 90 degrees along phi, and n3 is the # cross product of the two n1x,n1y,n1z = sin(phi)*cos(theta), sin(phi)*sin(theta), cos(phi) n2x,n2y,n2z = cos(phi)*cos(theta), cos(phi)*sin(theta), -sin(phi) # Cross product n3x,n3y,n3z = (n1y*n2z - n1z*n2y, n1z*n2x - n1x*n2z, n1x*n2y - n1y*n2x) # Convert pos from x/y/z coordinates to n1/n2/n3 coordinates u = n1x*x + n1y*y + n1z*z v = n2x*x + n2y*y + n2z*z w = n3x*x + n3y*y + n3z*z # Pull the amplitude from the shuffled array index ("j"), not "i", # otherwise neighboring unit vectors will have similar amplitudes! size = self.sizes[j] # Noise is the average of cosine of distance along # each axis, shifted by offsets and scaled by amplitude. noise += size/3*(cos(u/size + self.offsets[3*i]) + cos(v/size + self.offsets[3*i+1]) + cos(w/size + self.offsets[3*i+2])) end # Approximate normal CDF: noise /= 3*sigma return (0.5*(-1 if noise < 0 else 1) *sqrt(1 - exp(-2/pi * noise*noise)) + 0.5)

Whew! That’s a heck of a function! But look at the great results!

(The sine wave sizes here are generated by ((i+0.5)/(N+1))^-log(smoothness), where smoothness is an adjustable parameter.)

With all of these changes, the resulting Hill Noise algorithm:

- Will (practically) never repeat itself.
- Has highly configurable characteristics that are easy to understand (sine wave counts and size distributions).
- Is memory efficient and easy to implement in a shader, even without access to a pseudorandom number generator.
- Produces noise that is (nearly) uniformly distributed across the [0,1] interval.
- Can be modified to calculate exact surface normals/gradients.

I hope you enjoyed this post and find this noise function to be useful!

]]>But they can also be used to visualize functions that map (x,y) values to z values, or to find the boundaries of implicit surfaces. One of the common algorithms for finding contour lines is Marching Squares. The algorithm assumes that you have a set of elevation data points spaced out at regular intervals on a grid, and generates contour lines for each square of the grid. A simpler algorithm exists, called Meandering Triangles, and it divides the data points into triangles, rather than squares, to reduce the number of cases the algorithm has to handle. Here, I’ll give a full explanation of how the Meandering Triangles algorithm works, with some examples to show it in action.

First, let’s see how it looks:

In order to generate contour lines, we need some data whose contours we want to find.

import math # some arbitrary function that maps (x,y) to an elevation between 0 and 1 def elevation_function(x,y): return 1/(2+math.sin(2*math.sqrt(x*x + y*y))) * (.75+.5*math.sin(x*2)) elevation_data = dict() WIDTH, HEIGHT = 100, 100 SPACING = 1 for x in range(0,width, SPACING): for y in range(0,height, SPACING): elevation_data[(x,y)] = elevation_function(x,y)

The first step is to divide the regularly spaced grid of elevation data into triangles. In this example, I divide each set of 4 adjacent points into the top-left and bottom-right halves of a square, which forms two triangles.

import collections Triangle = collections.namedtuple("Triangle", "v1 v2 v3") triangles = [] for x in range(0,width-1, SPACING): for y in range(0,height-1, SPACING): t1 = Triangle((x,y), (x+SPACING,y), (x,y+SPACING)) triangles.append(t1) t2 = Triangle((x+SPACING,y), (x,y+SPACING), (x+SPACING,y+SPACING)) triangles.append(t2)

If we assume the elevation data is sampled at a high enough resolution, then each triangle is pretty close to a small, nearly-flat bit of terrain. If we’re drawing a contour line at elevation = 0.5, then the triangle will have a contour line going through it if some part of that triangle is below 0.5, and some part of it is above 0.5. The same is true for any threshold, but let’s assume we want to draw the contour line for 0.5 in this example. There are four possible cases here:

* All the vertices are below the threshold (no contour line)

* All the vertices are above the threshold (no contour line)

* Two of the vertices are below, and one is above the threshold (has a contour line)

* One of the vertices is below, and two are above the threshold (has a contour line)

threshold = 0.5 Edge = collections.namedtuple("Edge", "e1 e2") contour_segments = [] for triangle in triangles: below = [v for v in triangle if elevation_data[v] < threshold] above = [v for v in triangle if elevation_data[v] >= threshold] # All above or all below means no contour line here if len(below) == 0 or len(above) == 0: continue # We have a contour line, let's find it

Once we know that a contour line passes through a triangle, we also know that it passes between the vertices that are above the threshold and those that are below the threshold. Since it’s a triangle, we have 1 vertex on one side, and 2 on the other side, and the contour line will pass through 2 of the 3 edges of the triangle. Where along that edge the contour line will cross can be determined by linearly interpolating between the triangle vertices to find where the threshold’s exact value should fall.

threshold = 0.5 Edge = collections.namedtuple("Edge", "e1 e2") contour_segments = [] for triangle in triangles: below = [v for v in triangle if elevation_data[v] < threshold] above = [v for v in triangle if elevation_data[v] >= threshold] # All above or all below means no contour line here if len(below) == 0 or len(above) == 0: continue # We have a contour line, let's find it minority = above if len(above) < len(below) else below majority = above if len(above) > len(below) else below contour_points = [] crossed_edges = (Edge(minority[0], majority[0]), Edge(minority[0], majority[1])) for triangle_edge in crossed_edges: # how_far is a number between 0 and 1 indicating what percent # of the way along the edge (e1,e2) the crossing point is e1, e2 = triangle_edge.e1, triangle_edge.e2 how_far = ((threshold - elevation_data[e2]) / (elevation_data[e1] - elevation_data[e2])) crossing_point = ( how_far * e1[0] + (1-how_far) * e2[0], how_far * e1[1] + (1-how_far) * e2[1]) contour_points.append(crossing_point) contour_segments.append(Edge(contour_points[0], contour_points[1]))

All that’s left to do is join up all the adjacent line segments into lines. My implementation builds up contour lines by adding segments to the head and tail of a line until it runs out of connected segments.

unused_segments = set(contour_segments) segments_by_point = collections.defaultdict(set) for segment in contour_segments: segments_by_point[segment.e1].add(segment) segments_by_point[segment.e2].add(segment) contour_lines = [] while unused_segments: # Start with a random segment line = collections.deque(unused_segments.pop()) while True: tail_candidates = segments_by_point[line[-1]] & unused_segments if tail_candidates: tail = tail_candidates.pop() line.append(tail.e1 if tail.e2 == line[-1] else tail.e2) unused_segments.remove(tail) head_candidates = segments_by_point[line[0]] & unused_segments if head_candidates: head = head_candidates.pop() line.appendleft(head.e1 if head.e2 == line[0] else head.e2) unused_segments.remove(head) if not tail_candidates and not head_candidates: # There are no more segments touching this line, # so we're done with it. contour_lines.append(list(line)) break

And that’s it! The example code I’ve shown here assumes a single fixed threshold, but it’s easy to run multiple passes to find contour lines at multiple different thresholds. I think this algorithm is quite elegant in its simplicity, and it runs quickly on arbitrary data. There are certainly better algorithms if you care about the level of detail and have an elevation function that is slow to compute, but this algorithm makes an excellent first draft for finding contour lines.

''' Copyright © 2017 Bruce Hill <bruce@bruce-hill.com> This work is free. You can redistribute it and/or modify it under the terms of the Do What The Fuck You Want To Public License, Version 2, as published by Sam Hocevar. See http://www.wtfpl.net/ for more details. ''' import math import collections Triangle = collections.namedtuple("Triangle", "v1 v2 v3") Edge = collections.namedtuple("Edge", "e1 e2") def find_contours(elevation_function, xmin, xmax, ymin, ymax, spacing=1, elevation=0.5): ''' Return a list of contour lines that have a constant elevation near the specified value. Lines will be truncated to the sepcified x/y range, and the elevation function will be sampled at even intervals determined by the spacing parameter. ''' elevation_data = dict() for x in range(xmin, xmax+1, spacing): for y in range(ymin, ymax+1, spacing): elevation_data[(x,y)] = elevation_function(x,y) triangles = [] for x in range(xmin, xmax, spacing): for y in range(ymin, ymax, spacing): triangles.append(Triangle((x,y), (x+spacing,y), (x,y+spacing))) triangles.append(Triangle((x+spacing,y), (x,y+spacing), (x+spacing,y+spacing))) contour_segments = [] for triangle in triangles: below = [v for v in triangle if elevation_data[v] < elevation] above = [v for v in triangle if elevation_data[v] >= elevation] # All above or all below means no contour line here if len(below) == 0 or len(above) == 0: continue # We have a contour line, let's find it minority = above if len(above) < len(below) else below majority = above if len(above) > len(below) else below contour_points = [] crossed_edges = (Edge(minority[0], majority[0]), Edge(minority[0], majority[1])) for triangle_edge in crossed_edges: # how_far is a number between 0 and 1 indicating what percent of the way # along the edge (e1,e2) the crossing point is how_far = ((elevation - elevation_data[triangle_edge.e2]) / (elevation_data[triangle_edge.e1] - elevation_data[triangle_edge.e2])) crossing_point = ( how_far * triangle_edge.e1[0] + (1-how_far) * triangle_edge.e2[0], how_far * triangle_edge.e1[1] + (1-how_far) * triangle_edge.e2[1]) contour_points.append(crossing_point) contour_segments.append(Edge(contour_points[0], contour_points[1])) unused_segments = set(contour_segments) segments_by_point = collections.defaultdict(set) for segment in contour_segments: segments_by_point[segment.e1].add(segment) segments_by_point[segment.e2].add(segment) contour_lines = [] while unused_segments: # Start with a random segment line = collections.deque(unused_segments.pop()) while True: tail_candidates = segments_by_point[line[-1]].intersection(unused_segments) if tail_candidates: tail = tail_candidates.pop() line.append(tail.e1 if tail.e2 == line[-1] else tail.e2) unused_segments.remove(tail) head_candidates = segments_by_point[line[0]].intersection(unused_segments) if head_candidates: head = head_candidates.pop() line.appendleft(head.e1 if head.e2 == line[0] else head.e2) unused_segments.remove(head) if not tail_candidates and not head_candidates: # There are no more segments touching this line, so we're done with it. contour_lines.append(list(line)) break return contour_lines if __name__ == "__main__": # Run a test case with some arbitrary function # that maps (x,y) to an elevation between 0 and 1 def elevation_function(x,y): return 1/(2+math.sin(2*math.sqrt(x*x + y*y))) * (.75+.5*math.sin(x*2)) print(find_contours(elevation_function, 0,100, 0,100, spacing=1, elevation=0.5))]]>

Consider this problem: You have a list of items and a list of their weights. You want to randomly choose an item, with probability proportional to the item’s weight. For these examples, the algorithms will take a list of weights and return the index of the randomly selected item.

The simplest algorithm works by generating a random number and walking along the list of weights in a linear fashion. This has the advantage of requiring no additional space, but it runs in O(N) time, where N is the number of weights.

def weighted_random(weights): remaining_distance = random() * sum(weights) for i, weight in enumerate(weights): remaining_distance -= weight if remaining_distance < 0: return i

This algorithm can be improved slightly by sorting the weights in descending order so that the bigger weights will be reached more quickly, but this is only a constant speedup factor–the algorithm is still O(N) after the sorting has been done.

With a little bit of preprocessing, it’s possible to speed up the algorithm by storing the running totals of the weights, and using a binary search instead of a linear scan. This adds an additional storage cost of O(N), but it speeds the algorithm up to O(log(N)) for each random selection. Personally, I’m not a big fan of implementing binary search algorithms, because it’s so easy to make off-by-one errors. This is a pretty dang fast algorithm though.

def prepare_binary_search(weights): # Computing the running totals takes O(N) time running_totals = list(itertools.accumulate(weights)) def weighted_random(): target_distance = random()*running_totals[-1] low, high = 0, len(weights) while low < high: mid = int((low + high) / 2) distance = running_totals[mid] if distance < target_distance: low = mid + 1 elif distance > target_distance: high = mid else: return mid return low return weighted_random

However, it’s possible to do even better by using the following algorithm that I’ve come up with. The algorithm works by first sorting the weights in descending order, then hopping along the list to find the selected element. The size of hops is calculated based on the invariant that all weights after the current position are not larger than the current weight. The algorithm tends to quickly hop over runs of weights with similar size, but it sometimes makes overly conservative hops when the weight size changes. Fortunately, when the weight size changes, it always gets smaller during the search, which means that there is a corresponding decrease in the probability that the search will need to progress further. This means that although the worst case scenario is a linear traversal of the whole list, the average number of iterations is much smaller than the average number of iterations for the binary search algorithm (for large lists).

def prepare_hopscotch_selection(weights): # This preparation will run in O(N*log(N)) time, # or however long it takes to sort your weights sorted_indices = sorted(range(len(weights)), reverse=True, key=lambda i:weights[i]) sorted_weights = [weights[s] for s in sorted_indices] running_totals = list(itertools.accumulate(sorted_weights)) def weighted_random(): target_dist = random()*running_totals[-1] guess_index = 0 while True: if running_totals[guess_index] > target_dist: return sorted_indices[guess_index] weight = sorted_weights[guess_index] # All weights after guess_index are <= weight, so # we need to hop at least this far to reach target_dist hop_distance = target_dist - running_totals[guess_index] hop_indices = 1 + int(hop_distance / weight) guess_index += hop_indices return weighted_random

Performing a good amortized analysis of this algorithm is rather difficult, because its runtime depends on the distribution of the weights. The two most extreme cases I can think of are if the weights are all exactly the same, or if the weights are successive powers of 2.

If all weights are equal, then the algorithm always terminates within 2 iterations. If it needs to make a hop, the hop will go to exactly where it needs to. Thus, the runtime is O(1) in all cases.

If the weights are successive powers of 2 (e.g. [32,16,8,4,2,1]), then if the algorithm generates a very high random number, it might traverse the whole list of weights, one element at a time. This is because every weight is strictly greater than the sum of all smaller weights, so the calculation guesses it will only need to hop one index. However, it’s important to remember that this is not a search algorithm, it’s a weighted random number generator, which means that *each successive item is half as likely to be chosen as the previous one,* so across k calls, the average number of iterations will be <2k. In other words, as the time to find an element increases linearly, the probability of selecting that element decreases exponentially, so the amortized runtime in this case works out to be O(1). If the weights are powers of some number larger than 2, then the search time is still the same, but it's *even less likely* to need to walk to the later elements. If the weights are powers of some number between 1 and 2, then the algorithm will be able to jump ahead multiple elements at a time, since the sum of remaining weights won’t necessarily be smaller than the current weight.

These are the two most extreme cases I could think of to analyze, and the algorithm runs in amortized constant time for both. The algorithm quickly jumps across regions with similar-sized weights, and slowdowns caused by variance in weight sizes are offset by the fact that bigger weights near the front of the list (and hence faster) are more likely to be selected when there’s more variance in the weight sizes. My hunch is that this algorithm runs in amortized constant time for any weight distribution, but I’ve been unable to prove or disprove that hunch.

Here are some tests that show how the different algorithms perform as the number of weights increases, using different weight distributions. The graphs use a logarithmic x-axis, and you can see that the binary search algorithm is roughly a straight line, corresponding to its O(log(N)) runtime, and the linear scan variants have an exponential curve, corresponding to O(N). The hopscotch algorithm looks to me as if it’s asymptotically approaching a constant value, which would correspond to O(1), though it’s hard to say. It certainly does appear to be sublogarithmic though.

Python’s random.random() generates numbers in the half-open interval [0,1), and the implementations here all assume that random() will never return 1.0 exactly. It’s important to be wary of things like Python’s random.uniform(a,b), which generates results in the closed interval [a,b], because this can break some of the implementations here.

The implementations here also assume that all weights are strictly greater than zero. If you might have zero-weight items, the algorithm needs to be sure to never return those items, even if random() returns some number that exactly lines up with one of the running totals.

The version of the hopscotch algorithm I present here is the version that I initially conceived, and I think it’s the easiest to grok. However, if your weights do vary by many orders of magnitude, it is probably best to store the running totals from lowest to highest weight, to minimize floating point errors. It’s best to add small numbers together until the running total gets big, and then start adding big numbers to that, rather than adding very small numbers to a big running total. This requires adjusting the algorithm somewhat, but if you’re worried about floating point errors, then you’re probably more than capable of tweaking the code.

If anyone is able to do a good amortized analysis of this algorithm’s runtime for arbitrary weights, I’d love to hear about it. The algorithm is simple to implement and it seems to run much faster than O(log(N)). My gut says that it runs in amortized O(1) time, but you can’t always trust your gut.

There’s one more weighted random algorithm, described in this excellent page by Keith Schwarz, that I think is the fastest and most efficient algorithm out there. It’s really elegant, and it runs in O(N) setup time, and guaranteed O(1) run time for every random selection. The algorithm is hard to find on Google, despite being relatively simple to implement and clearly better than all other contenders (alas, including my own algorithm), so here is my Python implementation:

def prepare_aliased_randomizer(weights): N = len(weights) avg = sum(weights)/N aliases = [(1, None)]*N smalls = ((i, w/avg) for i,w in enumerate(weights) if w < avg) bigs = ((i, w/avg) for i,w in enumerate(weights) if w >= avg) small, big = next(smalls, None), next(bigs, None) while big and small: aliases[small[0]] = (small[1], population[big[0]]) big = (big[0], big[1] - (1-small[1])) if big[1] < 1: small = big big = next(bigs, None) else: small = next(smalls, None) def weighted_random(): r = random()*N i = int(r) odds, alias = aliases[i] return alias if (r-i) > odds else population[i] return weighted_random

For now, I’ll refrain from a full explanation, but essentially, the algorithm works by dividing the weights into bigger-than-average and smaller-than-average, and creating average-weight-sized buckets that each have a full small weight, and enough of one of the big weights to fill the rest of the bucket. The big weights are gradually whittled down in this fashion until they become smaller-than-average chunks themselves, and get put in a bucket with one of the remaining big chunks. The result is N buckets, each of which correspond to exactly one or two items. The bucket remembers what percent of the bucket belongs to each of its owners, and what the two owners are. Then, you can do a weighted random selection by generating a uniform float between 0 and N, and casting to an integer to find the bucket, and using the remainder to determine which of the bucket’s owners has been selected. Very clever! Again, please check out and share the more detailed explanation.

]]>