{"version":1,"lectureId":"01M14TX4X2AEBCN974JH81ZCHZ","attempt":0,"publication":{"slug":"bloom-filters","title":"The Bloom Filter","subject":"computer-science","summary":"A working engineer's introduction to the Bloom filter. It starts with the membership problem when the set of seen items is far too large to keep, then builds the structure a bit at a time: one row of bits, k hash functions, and three items inserted so that two of them end up sharing a bit. A fourth item that nobody inserted then comes back as present, because three unrelated items happened to cover its pattern. From there the false positive rate is derived from the number of hash functions, the size of the bit array and the number of insertions, and the trade off is drawn as a curve with a visible optimum rather than quoted. It closes on why a bit may never be cleared, and on a crawler that keeps a billion visited URLs in a gigabyte and a quarter.","metaDescription":"How a Bloom filter tests membership in a few bits per item, why false positives happen, how to pick k, and why deletion is forbidden.","transcript":"Here is a problem that turns up in almost every large system. Items arrive one at a time, and for each one you have to answer a single question. Have I seen this one before? A crawler asks it of every link it discovers, and a cache asks it of every key. So here is the question this lecture answers. Have I seen this item before, when the set of items I have seen is far too large to keep? The second half of that sentence is where all of the difficulty lives. The obvious answer is to keep the set. Every item you have seen goes into a hash set, and every new one gets looked up. Across the bottom here is n, the number of items, counted in millions. Up the side is memory, in gigabytes. URLs are not small. Sixty four bytes each is a kind estimate, and this red line is what that costs you. A billion of them is sixty four gigabytes, and it is a straight line, because an exact set charges you for every item, forever. The dashed line is the memory the machine actually has. Eight gigabytes. The red line crosses it here, at about a hundred and twenty five million URLs, and past that point the structure does not fit and the process dies. The blue line is where we are going. The same billion URLs, about ten bits each, and the whole structure is a gigabyte and a quarter. Fifty times smaller, and it never stores a single URL. You do not get that for free. What you give up is certainty, in one direction only. A Bloom filter answers two ways: definitely not in the set, or probably in the set. The second of those is sometimes wrong. Two things to hold on to before we build it. It never stores the items, so you can never ask it what is in there. And the mistakes only ever go one way. It will never tell you an item is absent when you did in fact insert it. So we are not going to store the items. We are going to store a fixed row of bits, and nothing else. Here is one: sixteen bits, numbered zero to fifteen, and every single one of them is zero. Call the length m. In a real filter m is not sixteen, it is a few billion. Sixteen is a size you can see. And notice what is missing here. There is no list of items anywhere in this picture, no copy of anything that was ever inserted. There are only bits. So take an item. Call it x. We hash it, not once, but k times, with k independent hash functions. Here k is three, and the first of them hands back position two. The second gives seven, and the third gives eleven. Three positions, three bits, and we set all three of them to one. That is the whole of the insert operation. Notice what did not happen there. We did not store x. We did not store a hash of x, or a pointer to x, or how long it was. Three bits went from zero to one, and the item itself is gone. Insert a second item. Its three hashes come out four, seven, and thirteen, so we set those. And look at position seven. It was already a one, set by the first item, and it stays a one. The row has no idea that two different items are now leaning on that single bit. Hold on to that. A third item, and its positions are one, nine, and fourteen. Three items are in the filter now, and eight of the sixteen bits are ones. That is the entire state. So now the other half of the job. How do you ask a question? To test an item you hash it exactly the same way, and you look at the k bits it names. Here is the second item again. Four, seven, thirteen. All three of them are ones. So the filter says: probably present. And it is right, we did insert that one. But look at what it actually checked. It never compared the item to anything at all. It read three bits. Now an item that was never inserted. Its positions come out zero, nine, and fourteen. The bit at nine is a one, the bit at fourteen is a one, and the bit at zero is not. And that settles it. If this item had ever been inserted, every one of its bits would have been set at that moment, and bits in this row are never cleared. A single zero anywhere is proof of absence. Definitely not in the set. That answer is exact, and it is the one guarantee the structure gives you. It is worth more than it sounds. Most of the time the thing you are asking about really is new, and you get told so for three bit lookups and no disk at all. Here is that same row, with those same three items in it, and now I am going to ask about an item that nobody ever inserted. Call it v. Its three positions come out two, four, and nine. Two is a one. The first item set it. Four is a one. The second item set that. Nine is a one, and the third item set that. So all k bits are ones, and the filter says: probably present. And it is wrong. No item set that pattern. Three different items each happened to set one of its bits, and between them they covered it. That is a false positive. It is not a bug, and there is no clever fix for it. It is what you bought when you decided to store bits instead of items. The bits do not remember who set them. What you can do is work out how often it happens, before you ship anything, and then choose how often you are willing to let it happen. So let us count it. Fix one bit in the row and follow it. Call P nought the chance that this bit is still a zero, and P one the chance that something has set it. A single hash function picks a position uniformly out of m, so it misses our bit with probability one minus one over m. Inserting one item runs k hashes. Inserting n items runs k times n of them, and if the hash functions are independent our bit survives every one of them, with probability one minus one over m, all raised to the power k n. That form is awkward to think with, so use the standard approximation. When m is large it is very close to e to the minus k n over m. Same number, and far easier to see what it does. Turn it around. The chance that a given bit has been set is one minus that. And this curve is exactly that number, for our sixteen bit row with three hash functions, as n climbs. With three items in, the formula says about forty three percent of the bits are ones. We counted eight out of sixteen, which is fifty. With only sixteen bits that is the sort of luck you get. Now push n up to twelve. Nearly ninety percent of the row is ones, and at that point almost any pattern you ask about is already covered by something. The false positive rate climbs with n, and this is the reason why. So now the query itself. An item that was never inserted names k bits. Each one is a one with probability P one, and if we treat those as independent, all k of them are ones with probability P one to the k. There it is. The false positive rate, epsilon, is one minus e to the minus k n over m, all raised to the k. Read the exponent as the total number of bits you have set, k of them for each of n items, divided by the room you gave them to spread out in. Read the rest of it the way an engineer reads it. n is handed to you by the problem. m is what you are willing to pay for. And k is the one you actually get to choose, and it is not at all obvious which way to push it. So, choosing k. Fix the budget at ten bits an item, which is the number you will see quoted everywhere, and the rate depends on k and nothing else. Here it is, plotted against the number of hash functions. With one hash function, each item sets a single bit, and the rate is about nine and a half percent. Nearly one query in ten comes back wrong. Two hash functions: three point three percent. Three: one point seven. Every extra hash function is another bit that has to be a one by accident, and that gets rapidly less likely. Five hash functions gets you under one percent. Seven gets you eight in a thousand, and that is the best this budget can do. Now watch what happens if you keep going. Ten hash functions is worse than seven. Fourteen is worse than three. The curve turns around and climbs back up. And the reason is sitting in the formula. Every extra hash function is one more bit a query has to match, which helps you. But it is also one more bit set on every insert, so the exponent grows and the row fills up faster, which hurts you. Past a point the second effect wins. The minimum is where those two balance, and it lands somewhere clean. The best k is m over n times the natural log of two, which for ten bits an item is six point nine. So you use seven. And there is a check you can run in production. At the optimal k the row ends up with almost exactly half its bits set. Half ones, half zeros. If your filter is much fuller than half, you are running too many hash functions. Which leaves the budget itself. Substitute that optimal k back into the rate, and the whole expression collapses into one number raised to the bits per item. About zero point six two, to the power m over n. So here is the curve an engineer actually wants. Bits per item across the bottom, false positive rate up the side. At eight bits an item, two percent. At ten bits, eight in a thousand. At fourteen, about one in a thousand. At twenty bits an item you are down to seven in a hundred thousand, and you are still using less than three bytes a URL. Every extra bit per item multiplies the error by roughly zero point six two. That is what makes ten the number people quote. It is the first place the curve is properly flat, and it is under one percent. And look at the shape of it. Going from three bits an item to eight is cheap, and it buys you an enormous amount. After that it gets expensive, because you are paying linearly for an error that is shrinking exponentially. One more property, and it is the one that catches people out. You cannot remove anything from a Bloom filter. Here is why, on the row we already have, with the same three items in it. Suppose we want to take the first item back out. Its bits were two, seven, and eleven, so the obvious move is to clear those three. Now ask about the second item, which is still very much in the set. Its bits are four, seven, and thirteen. Four is a one. Thirteen is a one. And seven is now a zero, because we just cleared it. So the filter says definitely absent, about an item that is definitely present. That is a false negative, and it is the single answer this structure promised you it would never get wrong. The bit at seven was never the first item's bit to give back. It was shared, and nothing in the row records how many items are leaning on it. So we put it back, and we do not do that again. Which gives us the rule, and it is worth stating plainly. A Bloom filter supports insert, and it supports query, and it does not support delete. If the things you are tracking ever go away again, this is the wrong structure for them. There is a variant that fixes it. Replace each bit with a small counter, increment on insert, decrement on delete, and now the shared position knows that two items are leaning on it. You pay for that. Four bit counters make the whole thing four times bigger, which was most of the reason you reached for a Bloom filter in the first place. So where does this earn its keep? A crawler is the clean case. You pull a URL off the frontier, and before you spend a network round trip on it, you ask whether you have fetched it already. The filter answers in the two ways we know. Definitely new, and you fetch it. Probably seen, and you drop it on the floor. A billion URLs at ten bits each is a gigabyte and a quarter, which fits in memory on any machine you would run a crawler on. The exact set was sixty four gigabytes, and a disk seek for every link on every page. At seven hash functions the rate is eight in a thousand, so about eight million URLs out of a billion get reported as seen when they never were, and never get fetched. On a web that size, that is a rounding error in your coverage. And notice which way the errors fall. What cannot happen is a page you have already fetched coming back as new. That is the one that puts a crawler in a loop, and it is precisely the answer the filter is exact about. So that is the whole structure. Bits instead of items. k hash functions instead of a comparison. A false positive rate you can compute before you ship. And no deletions, ever.","watch":{"version":1,"scenes":[{"title":"A Set You Cannot Afford to Keep","start":0,"end":125.44514583333334,"objects":{"bloom_line":"a FunctionPlot [blue] labelled \"upright(\"Bloom filter\")\" drawn in memory (function=<function>)","budget":"a Line [yellow] labelled \"upright(\"RAM\")\" drawn in memory (start=(0.0, 8.0), end=(1000.0, 8.0), dashed=True)","card":"a Title that says \"Data Structures for Systems — The Bloom Filter\"","cost":"a Table [text] that says \"Structure $10^9$ URLs Exact hash set 64 GB Bloom filter 1.25 GB\" (rows=(('Structure', '$10^9$ URLs'), ('Exact hash set', '64 GB'), ('B…, header=True)","exact_line":"a FunctionPlot [red] labelled \"upright(\"exact set\")\" drawn in memory (function=<function>)","memory":"an Axes (x_range=(0.0, 1000.0), y_range=(0.0, 70.0), x_ticks_every=200.0)","point":"a Point [yellow] drawn in memory (location=(125.0, 8.0))","point_2":"a Point [yellow] drawn in memory (location=(900.0, 1.125))","promise":"a Panel that says \"A Bloom filter answers two ways: definitely not in the set, or probably in the set. It never stores the items themselves.\"","question":"a Panel that says \"Have I seen this item before, when the set of items I have seen is far too large to keep?\""},"beats":[{"start":0,"say":"Here is a problem that turns up in almost every large system. Items arrive one at a time, and for each one you have to answer a single question. Have I seen this one before? A crawler asks it of every link it discovers, and a cache asks it of every key.","live":[],"does":[[0,"card is shown on the screen, written out."],[1.5,"card: enter:write-left-to-right."],[17.322,"card is hidden from the screen — left the board."]]},{"start":18.522,"say":"So here is the question this lecture answers. Have I seen this item before, when the set of items I have seen is far too large to keep? The second half of that sentence is where all of the difficulty lives.","live":null,"does":[[19.695,"question is shown on the screen, written out."],[31.0615,"question moves to a new place on the board."]]},{"start":31.661499999999997,"say":"The obvious answer is to keep the set. Every item you have seen goes into a hash set, and every new one gets looked up. Across the bottom here is n, the number of items, counted in millions. Up the side is memory, in gigabytes.","live":["question"],"does":[[31.661499999999997,"memory is shown on the screen, written out."]]},{"start":47.702,"say":"URLs are not small. Sixty four bytes each is a kind estimate, and this red line is what that costs you. A billion of them is sixty four gigabytes, and it is a straight line, because an exact set charges you for every item, forever.","live":["question","memory"],"does":[[53.27499999999999,"exact_line is shown on the screen, drawn."],[55.876,"memory moves to a new place on the board."],[55.876,"cost is shown on the screen, written out."],[56.275999999999996,"cost is shown on the screen, written out."]]},{"start":64.3015,"say":"The dashed line is the memory the machine actually has. Eight gigabytes. The red line crosses it here, at about a hundred and twenty five million URLs, and past that point the structure does not fit and the process dies.","live":["question","memory","exact_line"],"does":[[64.893,"budget is shown on the screen, written out."],[71.58100000000002,"point is shown on the screen, grown."],[73.58100000000002,"point is hidden from the screen."]]},{"start":80.575,"say":"The blue line is where we are going. The same billion URLs, about ten bits each, and the whole structure is a gigabyte and a quarter. Fifty times smaller, and it never stores a single URL.","live":["question","memory","exact_line","budget"],"does":[[81.30600000000001,"bloom_line is shown on the screen, drawn."],[88.94600000000001,"cost is shown on the screen, written out."]]},{"start":95.01400000000001,"say":"You do not get that for free. What you give up is certainty, in one direction only. A Bloom filter answers two ways: definitely not in the set, or probably in the set. The second of those is sometimes wrong.","live":["question","memory","exact_line","budget","bloom_line"],"does":[[102.47900000000001,"promise is shown on the screen, written out."]]},{"start":111.03200000000001,"say":"Two things to hold on to before we build it. It never stores the items, so you can never ask it what is in there. And the mistakes only ever go one way. It will never tell you an item is absent when you did in fact insert it.","live":["question","promise","memory","exact_line","budget","bloom_line"],"does":[[115.00200000000001,"promise (the \"never stores the items\" part) is emphasized."],[118.39200000000001,"promise (the \"never stores the items\" part) is no longer emphasized."],[121.99100000000001,"point_2 is shown on the screen, grown."],[123.99100000000001,"point_2 is hidden from the screen."],[124.40347916666667,"cost is hidden from the screen — left the board."],[124.40347916666667,"memory is hidden from the screen — left the board."],[124.40347916666667,"exact_line is hidden from the screen — memory left the board."],[124.40347916666667,"budget is hidden from the screen — memory left the board."],[124.40347916666667,"bloom_line is hidden from the screen — memory left the board."],[124.40347916666667,"promise is hidden from the screen — left the board."],[124.40347916666667,"question is hidden from the screen — left the board."]]}]},{"title":"Setting the Bits","start":125.44514583333334,"end":305.84783333333337,"objects":{"bits":"a Figure (x_range=(-0.5, 16.5), y_range=(-1.6, 1.6), aspect=(17.0, 3.2))","cells":"a Polygon [gray] drawn in bits (vertices=((0.0, 0.0), (1.0, 0.0), (1.0, 1.0), (0.0, 1.0)), filled=False)","cells_10":"a Polygon [gray] drawn in bits (vertices=((9.0, 0.0), (10.0, 0.0), (10.0, 1.0), (9.0, 1.0)), filled=False)","cells_11":"a Polygon [gray] drawn in bits (vertices=((10.0, 0.0), (11.0, 0.0), (11.0, 1.0), (10.0, 1.0)), filled=False)","cells_12":"a Polygon [gray] drawn in bits (vertices=((11.0, 0.0), (12.0, 0.0), (12.0, 1.0), (11.0, 1.0)), filled=False)","cells_13":"a Polygon [gray] drawn in bits (vertices=((12.0, 0.0), (13.0, 0.0), (13.0, 1.0), (12.0, 1.0)), filled=False)","cells_14":"a Polygon [gray] drawn in bits (vertices=((13.0, 0.0), (14.0, 0.0), (14.0, 1.0), (13.0, 1.0)), filled=False)","cells_15":"a Polygon [gray] drawn in bits (vertices=((14.0, 0.0), (15.0, 0.0), (15.0, 1.0), (14.0, 1.0)), filled=False)","cells_16":"a Polygon [gray] drawn in bits (vertices=((15.0, 0.0), (16.0, 0.0), (16.0, 1.0), (15.0, 1.0)), filled=False)","cells_2":"a Polygon [gray] drawn in bits (vertices=((1.0, 0.0), (2.0, 0.0), (2.0, 1.0), (1.0, 1.0)), filled=False)","cells_3":"a Polygon [gray] drawn in bits (vertices=((2.0, 0.0), (3.0, 0.0), (3.0, 1.0), (2.0, 1.0)), filled=False)","cells_4":"a Polygon [gray] drawn in bits (vertices=((3.0, 0.0), (4.0, 0.0), (4.0, 1.0), (3.0, 1.0)), filled=False)","cells_5":"a Polygon [gray] drawn in bits (vertices=((4.0, 0.0), (5.0, 0.0), (5.0, 1.0), (4.0, 1.0)), filled=False)","cells_6":"a Polygon [gray] drawn in bits (vertices=((5.0, 0.0), (6.0, 0.0), (6.0, 1.0), (5.0, 1.0)), filled=False)","cells_7":"a Polygon [gray] drawn in bits (vertices=((6.0, 0.0), (7.0, 0.0), (7.0, 1.0), (6.0, 1.0)), filled=False)","cells_8":"a Polygon [gray] drawn in bits (vertices=((7.0, 0.0), (8.0, 0.0), (8.0, 1.0), (7.0, 1.0)), filled=False)","cells_9":"a Polygon [gray] drawn in bits (vertices=((8.0, 0.0), (9.0, 0.0), (9.0, 1.0), (8.0, 1.0)), filled=False)","h1":"a Math [text] that says \"$h_1(x) = 2$\"","h2":"a Math [text] that says \"$h_2(x) = 7$\"","h3":"a Math [text] that says \"$h_3(x) = 11$\"","head_array":"a Heading that says \"The Whole Data Structure\"","head_insert":"a Heading that says \"Inserting an Item\"","head_query":"a Heading that says \"Asking a Question\"","ones":"a Polygon [blue] drawn in bits (vertices=((2.0, 0.0), (3.0, 0.0), (3.0, 1.0), (2.0, 1.0)), fill_opacity=0.75)","ones_2":"a Polygon [blue] drawn in bits (vertices=((7.0, 0.0), (8.0, 0.0), (8.0, 1.0), (7.0, 1.0)), fill_opacity=0.75)","ones_3":"a Polygon [blue] drawn in bits (vertices=((11.0, 0.0), (12.0, 0.0), (12.0, 1.0), (11.0, 1.0)), fill_opacity=0.75)","ones_4":"a Polygon [blue] drawn in bits (vertices=((4.0, 0.0), (5.0, 0.0), (5.0, 1.0), (4.0, 1.0)), fill_opacity=0.75)","ones_5":"a Polygon [blue] drawn in bits (vertices=((13.0, 0.0), (14.0, 0.0), (14.0, 1.0), (13.0, 1.0)), fill_opacity=0.75)","ones_6":"a Polygon [blue] drawn in bits (vertices=((1.0, 0.0), (2.0, 0.0), (2.0, 1.0), (1.0, 1.0)), fill_opacity=0.75)","ones_7":"a Polygon [blue] drawn in bits (vertices=((9.0, 0.0), (10.0, 0.0), (10.0, 1.0), (9.0, 1.0)), fill_opacity=0.75)","ones_8":"a Polygon [blue] drawn in bits (vertices=((14.0, 0.0), (15.0, 0.0), (15.0, 1.0), (14.0, 1.0)), fill_opacity=0.75)","point":"a Point [yellow] drawn in bits (location=(3.5, 0.5))","point_10":"a Point [yellow] drawn in bits (location=(9.5, 0.5))","point_11":"a Point [yellow] drawn in bits (location=(11.5, 0.5))","point_12":"a Point [yellow] drawn in bits (location=(13.5, 0.5))","point_13":"a Point [yellow] drawn in bits (location=(14.5, 0.5))","point_2":"a Point [yellow] drawn in bits (location=(11.5, 0.5))","point_3":"a Point [yellow] drawn in bits (location=(2.5, 0.5))","point_4":"a Point [yellow] drawn in bits (location=(7.5, 0.5))","point_5":"a Point [yellow] drawn in bits (location=(11.5, 0.5))","point_6":"a Point [yellow] drawn in bits (location=(1.5, 0.5))","point_7":"a Point [yellow] drawn in bits (location=(2.5, 0.5))","point_8":"a Point [yellow] drawn in bits (location=(4.5, 0.5))","point_9":"a Point [yellow] drawn in bits (location=(7.5, 0.5))","tags":"a Math [gray] that says \"$0$\" drawn in bits","tags_10":"a Math [gray] that says \"$9$\" drawn in bits","tags_11":"a Math [gray] that says \"$10$\" drawn in bits","tags_12":"a Math [gray] that says \"$11$\" drawn in bits","tags_13":"a Math [gray] that says \"$12$\" drawn in bits","tags_14":"a Math [gray] that says \"$13$\" drawn in bits","tags_15":"a Math [gray] that says \"$14$\" drawn in bits","tags_16":"a Math [gray] that says \"$15$\" drawn in bits","tags_2":"a Math [gray] that says \"$1$\" drawn in bits","tags_3":"a Math [gray] that says \"$2$\" drawn in bits","tags_4":"a Math [gray] that says \"$3$\" drawn in bits","tags_5":"a Math [gray] that says \"$4$\" drawn in bits","tags_6":"a Math [gray] that says \"$5$\" drawn in bits","tags_7":"a Math [gray] that says \"$6$\" drawn in bits","tags_8":"a Math [gray] that says \"$7$\" drawn in bits","tags_9":"a Math [gray] that says \"$8$\" drawn in bits","verdict":"a Math [text] that says \"$upright(\"probably present\")$\""},"beats":[{"start":125.44514583333334,"say":"So we are not going to store the items. We are going to store a fixed row of bits, and nothing else. Here is one: sixteen bits, numbered zero to fifteen, and every single one of them is zero. Call the length m.","live":[],"does":[[125.44514583333334,"head_array is shown on the screen, written out."],[125.44514583333334,"bits is shown on the screen, written out."],[129.30014583333335,"cells is shown on the screen, written out."],[129.35014583333333,"cells_2 is shown on the screen, written out."],[129.40014583333334,"cells_3 is shown on the screen, written out."],[129.45014583333335,"cells_4 is shown on the screen, written out."],[129.50014583333333,"cells_5 is shown on the screen, written out."],[129.55014583333335,"cells_6 is shown on the screen, written out."],[129.60014583333333,"cells_7 is shown on the screen, written out."],[129.65014583333334,"cells_8 is shown on the screen, written out."],[129.70014583333335,"cells_9 is shown on the screen, written out."],[129.75014583333333,"cells_10 is shown on the screen, written out."],[129.80014583333335,"cells_11 is shown on the screen, written out."],[129.85014583333333,"cells_12 is shown on the screen, written out."],[129.90014583333334,"cells_13 is shown on the screen, written out."],[129.95014583333335,"cells_14 is shown on the screen, written out."],[130.00014583333333,"cells_15 is shown on the screen, written out."],[130.05014583333335,"cells_16 is shown on the screen, written out."],[134.43114583333335,"tags is shown on the screen, written out."],[134.47114583333334,"tags_2 is shown on the screen, written out."],[134.51114583333333,"tags_3 is shown on the screen, written out."],[134.55114583333335,"tags_4 is shown on the screen, written out."],[134.59114583333334,"tags_5 is shown on the screen, written out."],[134.63114583333333,"tags_6 is shown on the screen, written out."],[134.67114583333336,"tags_7 is shown on the screen, written out."],[134.71114583333335,"tags_8 is shown on the screen, written out."],[134.75114583333334,"tags_9 is shown on the screen, written out."],[134.79114583333333,"tags_10 is shown on the screen, written out."],[134.83114583333335,"tags_11 is shown on the screen, written out."],[134.87114583333334,"tags_12 is shown on the screen, written out."],[134.91114583333334,"tags_13 is shown on the screen, written out."],[134.95114583333333,"tags_14 is shown on the screen, written out."],[134.99114583333335,"tags_15 is shown on the screen, written out."],[135.03114583333334,"tags_16 is shown on the screen, written out."]]},{"start":141.07964583333333,"say":"In a real filter m is not sixteen, it is a few billion. Sixteen is a size you can see. And notice what is missing here. There is no list of items anywhere in this picture, no copy of anything that was ever inserted. There are only bits.","live":["bits","head_array","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16"],"does":[[150.32114583333333,"point is shown on the screen, grown."],[152.31814583333335,"point_2 is shown on the screen, grown."],[152.32114583333333,"point is hidden from the screen."],[154.31814583333335,"point_2 is hidden from the screen."],[158.32064583333334,"bits moves to a new place on the board."],[158.32064583333334,"head_array is hidden from the screen — left the board."]]},{"start":158.92064583333334,"say":"So take an item. Call it x. We hash it, not once, but k times, with k independent hash functions. Here k is three, and the first of them hands back position two.","live":["bits","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16"],"does":[[158.92064583333334,"head_insert is shown on the screen, written out."],[169.95014583333335,"h1 is shown on the screen, written out."],[171.46014583333334,"ones is shown on the screen, faded in."]]},{"start":172.93064583333333,"say":"The second gives seven, and the third gives eleven. Three positions, three bits, and we set all three of them to one. That is the whole of the insert operation.","live":["bits","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16","h1","head_insert","ones"],"does":[[174.04514583333335,"h2 is shown on the screen, written out."],[174.04514583333335,"ones_2 is shown on the screen, faded in."],[175.58914583333336,"h3 is shown on the screen, written out."],[175.58914583333336,"ones_3 is shown on the screen, faded in."]]},{"start":184.52514583333334,"say":"Notice what did not happen there. We did not store x. We did not store a hash of x, or a pointer to x, or how long it was. Three bits went from zero to one, and the item itself is gone.","live":["bits","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16","h1","h2","h3","head_insert","ones","ones_2","ones_3"],"does":[[194.04514583333332,"point_3 is shown on the screen, grown."],[194.37014583333337,"point_4 is shown on the screen, grown."],[196.04514583333332,"point_3 is hidden from the screen."],[196.37014583333337,"point_4 is hidden from the screen."],[197.64414583333337,"point_5 is shown on the screen, grown."]]},{"start":198.99864583333334,"say":"Insert a second item. Its three hashes come out four, seven, and thirteen, so we set those.","live":["bits","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16","h1","h2","h3","head_insert","ones","ones_2","ones_3","point_5"],"does":[[199.64414583333337,"point_5 is hidden from the screen."],[202.47014583333336,"h1 becomes \"$h_1(y) = 4$\"."],[202.47014583333336,"ones_4 is shown on the screen, faded in."],[203.05014583333337,"h2 becomes \"$h_2(y) = 7$\"."],[203.67714583333336,"h3 becomes \"$h_3(y) = 13$\"."],[203.67714583333336,"ones_5 is shown on the screen, faded in."]]},{"start":206.07714583333336,"say":"And look at position seven. It was already a one, set by the first item, and it stays a one. The row has no idea that two different items are now leaning on that single bit. Hold on to that.","live":["bits","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16","h1","h2","h3","head_insert","ones","ones_2","ones_3","ones_4","ones_5"],"does":[[207.38914583333337,"The point (7.5, 0.5) in bits is lit up."],[217.16414583333335,"bits: retire a lit point (unemphasize_point)."]]},{"start":218.99514583333337,"say":"A third item, and its positions are one, nine, and fourteen.","live":null,"does":[[221.10814583333337,"h1 becomes \"$h_1(z) = 1$\"."],[221.10814583333337,"ones_6 is shown on the screen, faded in."],[221.6071458333334,"h2 becomes \"$h_2(z) = 9$\"."],[221.6071458333334,"ones_7 is shown on the screen, faded in."],[222.17614583333335,"h3 becomes \"$h_3(z) = 14$\"."],[222.17614583333335,"ones_8 is shown on the screen, faded in."]]},{"start":223.71714583333335,"say":"Three items are in the filter now, and eight of the sixteen bits are ones. That is the entire state. So now the other half of the job. How do you ask a question?","live":["bits","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16","h1","h2","h3","head_insert","ones","ones_2","ones_3","ones_4","ones_5","ones_6","ones_7","ones_8"],"does":[[226.10814583333337,"point_6 is shown on the screen, grown."],[226.10814583333337,"point_7 is shown on the screen, grown."],[226.10814583333337,"point_8 is shown on the screen, grown."],[226.10814583333337,"point_9 is shown on the screen, grown."],[226.10814583333337,"point_10 is shown on the screen, grown."],[226.10814583333337,"point_11 is shown on the screen, grown."],[226.10814583333337,"point_12 is shown on the screen, grown."],[226.10814583333337,"point_13 is shown on the screen, grown."],[227.60814583333337,"point_6 is hidden from the screen."],[227.60814583333337,"point_7 is hidden from the screen."],[227.60814583333337,"point_8 is hidden from the screen."],[227.60814583333337,"point_9 is hidden from the screen."],[227.60814583333337,"point_10 is hidden from the screen."],[227.60814583333337,"point_11 is hidden from the screen."],[227.60814583333337,"point_12 is hidden from the screen."],[227.60814583333337,"point_13 is hidden from the screen."],[235.43114583333335,"bits moves to a new place on the board."],[235.43114583333335,"h1 moves to a new place on the board."],[235.43114583333335,"h2 moves to a new place on the board."],[235.43114583333335,"h3 moves to a new place on the board."],[235.43114583333335,"head_insert is hidden from the screen — left the board."]]},{"start":236.03114583333337,"say":"To test an item you hash it exactly the same way, and you look at the k bits it names. Here is the second item again. Four, seven, thirteen. All three of them are ones.","live":["bits","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16","h1","h2","h3","ones","ones_2","ones_3","ones_4","ones_5","ones_6","ones_7","ones_8"],"does":[[236.03114583333337,"head_query is shown on the screen, written out."],[244.15814583333338,"h1 becomes \"$h_1(y) = 4$\"."],[244.15814583333338,"The point (4.5, 0.5) in bits is lit up."],[244.66914583333335,"h2 becomes \"$h_2(y) = 7$\"."],[244.66914583333335,"The point (7.5, 0.5) in bits is lit up."],[245.14514583333334,"h3 becomes \"$h_3(y) = 13$\"."],[245.14514583333334,"The point (13.5, 0.5) in bits is lit up."]]},{"start":249.08864583333337,"say":"So the filter says: probably present. And it is right, we did insert that one. But look at what it actually checked. It never compared the item to anything at all. It read three bits.","live":["bits","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16","h1","h2","h3","ones","ones_2","ones_3","ones_4","ones_5","ones_6","ones_7","ones_8","head_query"],"does":[[250.90014583333337,"verdict is shown on the screen, written out."]]},{"start":262.5406458333334,"say":"Now an item that was never inserted. Its positions come out zero, nine, and fourteen. The bit at nine is a one, the bit at fourteen is a one, and the bit at zero is not.","live":["bits","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16","h1","h2","h3","ones","ones_2","ones_3","ones_4","ones_5","ones_6","ones_7","ones_8","verdict","head_query"],"does":[[262.5406458333334,"bits: retire a lit point (unemphasize_point)."],[262.5406458333334,"bits: retire a lit point (unemphasize_point)."],[262.5406458333334,"bits: retire a lit point (unemphasize_point)."],[266.9291458333334,"h1 becomes \"$h_1(w) = 0$\"."],[267.5561458333334,"h2 becomes \"$h_2(w) = 9$\"."],[268.1951458333334,"h3 becomes \"$h_3(w) = 14$\"."],[270.3771458333334,"The point (9.5, 0.5) in bits is lit up."],[272.0371458333334,"The point (14.5, 0.5) in bits is lit up."],[274.0691458333334,"The point (0.5, 0.5) in bits is lit up."]]},{"start":276.09764583333333,"say":"And that settles it. If this item had ever been inserted, every one of its bits would have been set at that moment, and bits in this row are never cleared. A single zero anywhere is proof of absence. Definitely not in the set.","live":null,"does":[[288.82214583333337,"verdict becomes \"$upright(\"definitely absent\")$\"."]]},{"start":291.09364583333337,"say":"That answer is exact, and it is the one guarantee the structure gives you. It is worth more than it sounds. Most of the time the thing you are asking about really is new, and you get told so for three bit lookups and no disk at all.","live":null,"does":[[291.09364583333337,"bits: retire a lit point (unemphasize_point)."],[291.09364583333337,"bits: retire a lit point (unemphasize_point)."],[291.09364583333337,"bits: retire a lit point (unemphasize_point)."],[292.1501458333334,"A box is drawn around verdict."],[304.8061666666667,"bits is hidden from the screen — left the board."],[304.8061666666667,"cells is hidden from the screen — bits left the board."],[304.8061666666667,"cells_2 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_3 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_4 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_5 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_6 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_7 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_8 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_9 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_10 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_11 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_12 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_13 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_14 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_15 is hidden from the screen — bits left the board."],[304.8061666666667,"cells_16 is hidden from the screen — bits left the board."],[304.8061666666667,"tags is hidden from the screen — bits left the board."],[304.8061666666667,"tags_2 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_3 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_4 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_5 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_6 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_7 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_8 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_9 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_10 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_11 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_12 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_13 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_14 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_15 is hidden from the screen — bits left the board."],[304.8061666666667,"tags_16 is hidden from the screen — bits left the board."],[304.8061666666667,"ones is hidden from the screen — bits left the board."],[304.8061666666667,"ones_2 is hidden from the screen — bits left the board."],[304.8061666666667,"ones_3 is hidden from the screen — bits left the board."],[304.8061666666667,"ones_4 is hidden from the screen — bits left the board."],[304.8061666666667,"ones_5 is hidden from the screen — bits left the board."],[304.8061666666667,"ones_6 is hidden from the screen — bits left the board."],[304.8061666666667,"ones_7 is hidden from the screen — bits left the board."],[304.8061666666667,"ones_8 is hidden from the screen — bits left the board."],[304.8061666666667,"h1 is hidden from the screen — left the board."],[304.8061666666667,"h2 is hidden from the screen — left the board."],[304.8061666666667,"h3 is hidden from the screen — left the board."],[304.8061666666667,"head_query is hidden from the screen — left the board."],[304.8061666666667,"verdict is hidden from the screen — left the board."]]}]},{"title":"The False Positive","start":305.84783333333337,"end":501.55745833333333,"objects":{"bits":"a Figure (x_range=(-0.5, 16.5), y_range=(-1.6, 1.6), aspect=(17.0, 3.2))","cells":"a Polygon [gray] drawn in bits (vertices=((0.0, 0.0), (1.0, 0.0), (1.0, 1.0), (0.0, 1.0)), filled=False)","cells_10":"a Polygon [gray] drawn in bits (vertices=((9.0, 0.0), (10.0, 0.0), (10.0, 1.0), (9.0, 1.0)), filled=False)","cells_11":"a Polygon [gray] drawn in bits (vertices=((10.0, 0.0), (11.0, 0.0), (11.0, 1.0), (10.0, 1.0)), filled=False)","cells_12":"a Polygon [gray] drawn in bits (vertices=((11.0, 0.0), (12.0, 0.0), (12.0, 1.0), (11.0, 1.0)), filled=False)","cells_13":"a Polygon [gray] drawn in bits (vertices=((12.0, 0.0), (13.0, 0.0), (13.0, 1.0), (12.0, 1.0)), filled=False)","cells_14":"a Polygon [gray] drawn in bits (vertices=((13.0, 0.0), (14.0, 0.0), (14.0, 1.0), (13.0, 1.0)), filled=False)","cells_15":"a Polygon [gray] drawn in bits (vertices=((14.0, 0.0), (15.0, 0.0), (15.0, 1.0), (14.0, 1.0)), filled=False)","cells_16":"a Polygon [gray] drawn in bits (vertices=((15.0, 0.0), (16.0, 0.0), (16.0, 1.0), (15.0, 1.0)), filled=False)","cells_2":"a Polygon [gray] drawn in bits (vertices=((1.0, 0.0), (2.0, 0.0), (2.0, 1.0), (1.0, 1.0)), filled=False)","cells_3":"a Polygon [gray] drawn in bits (vertices=((2.0, 0.0), (3.0, 0.0), (3.0, 1.0), (2.0, 1.0)), filled=False)","cells_4":"a Polygon [gray] drawn in bits (vertices=((3.0, 0.0), (4.0, 0.0), (4.0, 1.0), (3.0, 1.0)), filled=False)","cells_5":"a Polygon [gray] drawn in bits (vertices=((4.0, 0.0), (5.0, 0.0), (5.0, 1.0), (4.0, 1.0)), filled=False)","cells_6":"a Polygon [gray] drawn in bits (vertices=((5.0, 0.0), (6.0, 0.0), (6.0, 1.0), (5.0, 1.0)), filled=False)","cells_7":"a Polygon [gray] drawn in bits (vertices=((6.0, 0.0), (7.0, 0.0), (7.0, 1.0), (6.0, 1.0)), filled=False)","cells_8":"a Polygon [gray] drawn in bits (vertices=((7.0, 0.0), (8.0, 0.0), (8.0, 1.0), (7.0, 1.0)), filled=False)","cells_9":"a Polygon [gray] drawn in bits (vertices=((8.0, 0.0), (9.0, 0.0), (9.0, 1.0), (8.0, 1.0)), filled=False)","defs":"a Text [text] that says \"Write $P_0$ for the chance that one chosen bit is still zero, and $P_1$ for the chance that something has set it.\"","fill":"an Axes (x_range=(0.0, 12.0), x_ticks_every=3.0, y_ticks_every=0.25)","head_fp":"a Heading that says \"An Item Nobody Inserted\"","head_math":"a Heading that says \"How Often Does That Happen?\"","n_live":"a VariableNumber (initial_value=3.0, format_spec='.0f')","ones":"a Polygon [blue] drawn in bits (vertices=((1.0, 0.0), (2.0, 0.0), (2.0, 1.0), (1.0, 1.0)), fill_opacity=0.75)","ones_2":"a Polygon [blue] drawn in bits (vertices=((2.0, 0.0), (3.0, 0.0), (3.0, 1.0), (2.0, 1.0)), fill_opacity=0.75)","ones_3":"a Polygon [blue] drawn in bits (vertices=((4.0, 0.0), (5.0, 0.0), (5.0, 1.0), (4.0, 1.0)), fill_opacity=0.75)","ones_4":"a Polygon [blue] drawn in bits (vertices=((7.0, 0.0), (8.0, 0.0), (8.0, 1.0), (7.0, 1.0)), fill_opacity=0.75)","ones_5":"a Polygon [blue] drawn in bits (vertices=((9.0, 0.0), (10.0, 0.0), (10.0, 1.0), (9.0, 1.0)), fill_opacity=0.75)","ones_6":"a Polygon [blue] drawn in bits (vertices=((11.0, 0.0), (12.0, 0.0), (12.0, 1.0), (11.0, 1.0)), fill_opacity=0.75)","ones_7":"a Polygon [blue] drawn in bits (vertices=((13.0, 0.0), (14.0, 0.0), (14.0, 1.0), (13.0, 1.0)), fill_opacity=0.75)","ones_8":"a Polygon [blue] drawn in bits (vertices=((14.0, 0.0), (15.0, 0.0), (15.0, 1.0), (14.0, 1.0)), fill_opacity=0.75)","probe":"a PlotPoint [yellow] labelled \"3\" drawn in fill (target='share', x=<VariableNumber n_live = 6.0>)","q1":"a Math [text] that says \"$h_1(v) = 2$\"","q2":"a Math [text] that says \"$h_2(v) = 4$\"","q3":"a Math [text] that says \"$h_3(v) = 9$\"","share":"a FunctionPlot [blue] drawn in fill (function=<function>)","tags":"a Math [gray] that says \"$0$\" drawn in bits","tags_10":"a Math [gray] that says \"$9$\" drawn in bits","tags_11":"a Math [gray] that says \"$10$\" drawn in bits","tags_12":"a Math [gray] that says \"$11$\" drawn in bits","tags_13":"a Math [gray] that says \"$12$\" drawn in bits","tags_14":"a Math [gray] that says \"$13$\" drawn in bits","tags_15":"a Math [gray] that says \"$14$\" drawn in bits","tags_16":"a Math [gray] that says \"$15$\" drawn in bits","tags_2":"a Math [gray] that says \"$1$\" drawn in bits","tags_3":"a Math [gray] that says \"$2$\" drawn in bits","tags_4":"a Math [gray] that says \"$3$\" drawn in bits","tags_5":"a Math [gray] that says \"$4$\" drawn in bits","tags_6":"a Math [gray] that says \"$5$\" drawn in bits","tags_7":"a Math [gray] that says \"$6$\" drawn in bits","tags_8":"a Math [gray] that says \"$7$\" drawn in bits","tags_9":"a Math [gray] that says \"$8$\" drawn in bits","verdict":"a Math [text] that says \"$upright(\"probably present\")$\"","work":"a Derivation [text] that says \"$P_0 &= (1 - 1 slash m)^(k n) \\ &approx e^(-k n slash m) \\ P_1 &approx 1 - e^(-k n slash m) \\ epsilon &approx (1 - e^(-k n slash m))^k$\""},"beats":[{"start":305.84783333333337,"say":"Here is that same row, with those same three items in it, and now I am going to ask about an item that nobody ever inserted. Call it v. Its three positions come out two, four, and nine.","live":[],"does":[[305.84783333333337,"head_fp is shown on the screen, written out."],[305.84783333333337,"bits is shown on the screen, written out."],[306.8228333333334,"cells is shown on the screen, faded in."],[306.86283333333336,"cells_2 is shown on the screen, faded in."],[306.9028333333334,"cells_3 is shown on the screen, faded in."],[306.9428333333334,"cells_4 is shown on the screen, faded in."],[306.98283333333336,"cells_5 is shown on the screen, faded in."],[307.0228333333334,"cells_6 is shown on the screen, faded in."],[307.06283333333334,"cells_7 is shown on the screen, faded in."],[307.10283333333336,"cells_8 is shown on the screen, faded in."],[307.1428333333334,"cells_9 is shown on the screen, faded in."],[307.18283333333335,"cells_10 is shown on the screen, faded in."],[307.22283333333337,"cells_11 is shown on the screen, faded in."],[307.2628333333334,"cells_12 is shown on the screen, faded in."],[307.30283333333335,"cells_13 is shown on the screen, faded in."],[307.3428333333334,"cells_14 is shown on the screen, faded in."],[307.3828333333334,"cells_15 is shown on the screen, faded in."],[307.42283333333336,"cells_16 is shown on the screen, faded in."],[308.49483333333336,"ones is shown on the screen, faded in."],[308.5448333333334,"ones_2 is shown on the screen, faded in."],[308.5948333333334,"ones_3 is shown on the screen, faded in."],[308.6448333333334,"ones_4 is shown on the screen, faded in."],[308.69483333333335,"ones_5 is shown on the screen, faded in."],[308.73483333333337,"tags is shown on the screen, written out."],[308.74483333333336,"ones_6 is shown on the screen, faded in."],[308.76483333333334,"tags_2 is shown on the screen, written out."],[308.7948333333334,"tags_3 is shown on the screen, written out."],[308.7948333333334,"ones_7 is shown on the screen, faded in."],[308.82483333333334,"tags_4 is shown on the screen, written out."],[308.8448333333334,"ones_8 is shown on the screen, faded in."],[308.8548333333334,"tags_5 is shown on the screen, written out."],[308.88483333333335,"tags_6 is shown on the screen, written out."],[308.9148333333334,"tags_7 is shown on the screen, written out."],[308.94483333333335,"tags_8 is shown on the screen, written out."],[308.9748333333334,"tags_9 is shown on the screen, written out."],[309.00483333333335,"tags_10 is shown on the screen, written out."],[309.0348333333334,"tags_11 is shown on the screen, written out."],[309.06483333333335,"tags_12 is shown on the screen, written out."],[309.0948333333334,"tags_13 is shown on the screen, written out."],[309.12483333333336,"tags_14 is shown on the screen, written out."],[309.1548333333334,"tags_15 is shown on the screen, written out."],[309.18483333333336,"tags_16 is shown on the screen, written out."],[315.93683333333337,"q1 is shown on the screen, written out."],[316.33183333333335,"q2 is shown on the screen, written out."],[316.7848333333334,"q3 is shown on the screen, written out."]]},{"start":318.01133333333337,"say":"Two is a one. The first item set it. Four is a one. The second item set that. Nine is a one, and the third item set that.","live":["bits","q1","q2","q3","head_fp","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","ones","ones_2","ones_3","ones_4","ones_5","ones_6","ones_7","ones_8","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16"],"does":[[318.2318333333334,"The point (2.5, 0.5) in bits is lit up."],[321.37783333333334,"The point (4.5, 0.5) in bits is lit up."],[325.02383333333336,"The point (9.5, 0.5) in bits is lit up."]]},{"start":328.6193333333334,"say":"So all k bits are ones, and the filter says: probably present. And it is wrong. No item set that pattern. Three different items each happened to set one of its bits, and between them they covered it.","live":null,"does":[[332.4508333333334,"verdict is shown on the screen, written out."]]},{"start":343.19733333333335,"say":"That is a false positive. It is not a bug, and there is no clever fix for it. It is what you bought when you decided to store bits instead of items. The bits do not remember who set them.","live":["bits","q1","q2","q3","verdict","head_fp","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","ones","ones_2","ones_3","ones_4","ones_5","ones_6","ones_7","ones_8","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16"],"does":[[344.4748333333334,"bits: retire a lit point (unemphasize_point)."],[344.4748333333334,"bits: retire a lit point (unemphasize_point)."],[344.4748333333334,"bits: retire a lit point (unemphasize_point)."]]},{"start":356.12683333333337,"say":"What you can do is work out how often it happens, before you ship anything, and then choose how often you are willing to let it happen. So let us count it.","live":null,"does":[[358.10083333333336,"verdict is indicated — a transient flash."],[366.1813333333333,"bits is hidden from the screen — left the board."],[366.1813333333333,"cells is hidden from the screen — bits left the board."],[366.1813333333333,"cells_2 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_3 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_4 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_5 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_6 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_7 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_8 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_9 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_10 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_11 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_12 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_13 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_14 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_15 is hidden from the screen — bits left the board."],[366.1813333333333,"cells_16 is hidden from the screen — bits left the board."],[366.1813333333333,"ones is hidden from the screen — bits left the board."],[366.1813333333333,"ones_2 is hidden from the screen — bits left the board."],[366.1813333333333,"ones_3 is hidden from the screen — bits left the board."],[366.1813333333333,"ones_4 is hidden from the screen — bits left the board."],[366.1813333333333,"ones_5 is hidden from the screen — bits left the board."],[366.1813333333333,"ones_6 is hidden from the screen — bits left the board."],[366.1813333333333,"ones_7 is hidden from the screen — bits left the board."],[366.1813333333333,"ones_8 is hidden from the screen — bits left the board."],[366.1813333333333,"tags is hidden from the screen — bits left the board."],[366.1813333333333,"tags_2 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_3 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_4 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_5 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_6 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_7 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_8 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_9 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_10 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_11 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_12 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_13 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_14 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_15 is hidden from the screen — bits left the board."],[366.1813333333333,"tags_16 is hidden from the screen — bits left the board."],[366.1813333333333,"head_fp is hidden from the screen — left the board."],[366.1813333333333,"q1 is hidden from the screen — left the board."],[366.1813333333333,"q2 is hidden from the screen — left the board."],[366.1813333333333,"q3 is hidden from the screen — left the board."],[366.1813333333333,"verdict is hidden from the screen — left the board."]]},{"start":367.3813333333334,"say":"Fix one bit in the row and follow it. Call P nought the chance that this bit is still a zero, and P one the chance that something has set it. A single hash function picks a position uniformly out of m, so it misses our bit with probability one minus one over m.","live":[],"does":[[367.3813333333334,"head_math is shown on the screen, written out."],[370.4458333333333,"defs is shown on the screen, written out."]]},{"start":384.14233333333334,"say":"Inserting one item runs k hashes. Inserting n items runs k times n of them, and if the hash functions are independent our bit survives every one of them, with probability one minus one over m, all raised to the power k n.","live":["defs","head_math"],"does":[[392.43283333333335,"work is shown on the screen, written out."]]},{"start":399.3833333333334,"say":"That form is awkward to think with, so use the standard approximation. When m is large it is very close to e to the minus k n over m. Same number, and far easier to see what it does.","live":null,"does":[[402.65683333333334,"work is shown on the screen, written out."]]},{"start":412.54533333333336,"say":"Turn it around. The chance that a given bit has been set is one minus that. And this curve is exactly that number, for our sixteen bit row with three hash functions, as n climbs.","live":null,"does":[[412.89383333333336,"work is shown on the screen, written out."],[417.8738333333333,"fill is shown on the screen, written out."],[417.8738333333333,"share is shown on the screen, drawn."]]},{"start":425.11483333333337,"say":"With three items in, the formula says about forty three percent of the bits are ones. We counted eight out of sixteen, which is fifty. With only sixteen bits that is the sort of luck you get. Now push n up to twelve.","live":["defs","fill","head_math","share"],"does":[[425.7418333333334,"probe is shown on the screen, written out."],[437.64183333333335,"probe is redrawn as the numbers it depends on change."],[437.64183333333335,"n_live ticks to 12.0."]]},{"start":439.77383333333336,"say":"Nearly ninety percent of the row is ones, and at that point almost any pattern you ask about is already covered by something. The false positive rate climbs with n, and this is the reason why.","live":["defs","fill","head_math","share","probe"],"does":[[448.9698333333333,"share is indicated — a transient flash."]]},{"start":452.57683333333335,"say":"So now the query itself. An item that was never inserted names k bits. Each one is a one with probability P one, and if we treat those as independent, all k of them are ones with probability P one to the k.","live":null,"does":[[463.66383333333334,"work is shown on the screen, written out."]]},{"start":467.73533333333336,"say":"There it is. The false positive rate, epsilon, is one minus e to the minus k n over m, all raised to the k. Read the exponent as the total number of bits you have set, k of them for each of n items, divided by the room you gave them to spread out in.","live":null,"does":[[470.7658333333334,"A box is drawn around work."],[478.76483333333334,"work (the \"k n\" part) is emphasized."],[484.23283333333336,"work (the \"k n\" part) is no longer emphasized."],[484.23283333333336,"work (the \"m\" part) is emphasized."],[486.2993333333334,"work (the \"m\" part) is no longer emphasized."]]},{"start":486.89933333333335,"say":"Read the rest of it the way an engineer reads it. n is handed to you by the problem. m is what you are willing to pay for. And k is the one you actually get to choose, and it is not at all obvious which way to push it.","live":null,"does":[[490.6488333333333,"probe is redrawn as the numbers it depends on change."],[490.6488333333333,"n_live ticks to 6.0."],[500.5157916666667,"defs is hidden from the screen — left the board."],[500.5157916666667,"fill is hidden from the screen — left the board."],[500.5157916666667,"share is hidden from the screen — fill left the board."],[500.5157916666667,"probe is hidden from the screen — fill left the board."],[500.5157916666667,"head_math is hidden from the screen — left the board."],[500.5157916666667,"work is hidden from the screen — left the board."]]}]},{"title":"Choosing the Hash Functions","start":501.55745833333333,"end":684.4156458333333,"objects":{"b_curve":"a FunctionPlot [green] drawn in b_plot (function=<function>)","b_live":"a VariableNumber (initial_value=8.0, format_spec='.0f')","b_plot":"an Axes (x_range=(2.0, 20.0), y_range=(0.0, 0.4), x_ticks_every=2.0)","b_probe":"a PlotPoint [yellow] labelled \"8\" drawn in b_plot (target='b_curve', x=<VariableNumber b_live = 20.0>)","budget_read":"a Math [text] that says \"$m slash n = 8: quad epsilon approx 0.021$\"","collapsed":"a Math [text] that says \"$epsilon approx 0.62^(m slash n)$\"","head_budget":"a Heading that says \"What Another Bit Buys You\"","head_k":"a Heading that says \"Ten Bits an Item: the Curve in $k$\"","k_curve":"a FunctionPlot [blue] drawn in k_plot (function=<function>)","k_live":"a VariableNumber (initial_value=1.0, format_spec='.0f')","k_plot":"an Axes (x_range=(1.0, 16.0), y_range=(0.0, 0.11), x_ticks_every=2.0)","k_probe":"a PlotPoint [yellow] labelled \"1\" drawn in k_plot (target='k_curve', x=<VariableNumber k_live = 14.0>)","optimum":"a Math [text] that says \"$k^* = frac(m, n) ln 2 approx 0.693 frac(m, n)$\"","point":"a Point [yellow] drawn in k_plot (location=(7.0, 0.0082))","point_2":"a Point [yellow] drawn in b_plot (location=(3.0, 0.2366))","point_3":"a Point [yellow] drawn in b_plot (location=(8.0, 0.0214))","rate":"a Math [text] that says \"$epsilon approx (1 - e^(-k n slash m))^k$\"","reading":"a Math [text] that says \"$k = 1: quad epsilon approx 0.095$\""},"beats":[{"start":501.55745833333333,"say":"So, choosing k. Fix the budget at ten bits an item, which is the number you will see quoted everywhere, and the rate depends on k and nothing else. Here it is, plotted against the number of hash functions.","live":[],"does":[[501.55745833333333,"head_k is shown on the screen, written out."],[508.73245833333334,"rate is shown on the screen, written out."],[512.5984583333334,"k_plot is shown on the screen, written out."],[512.5984583333334,"k_curve is shown on the screen, drawn."]]},{"start":515.7064583333333,"say":"With one hash function, each item sets a single bit, and the rate is about nine and a half percent. Nearly one query in ten comes back wrong.","live":["rate","k_plot","head_k","k_curve"],"does":[[516.2984583333333,"k_probe is shown on the screen, written out."],[520.6174583333334,"reading is shown on the screen, written out."]]},{"start":526.0704583333334,"say":"Two hash functions: three point three percent. Three: one point seven. Every extra hash function is another bit that has to be a one by accident, and that gets rapidly less likely.","live":["rate","reading","k_plot","head_k","k_curve","k_probe"],"does":[[526.4184583333333,"k_probe is redrawn as the numbers it depends on change."],[526.4184583333333,"reading becomes \"$k = 2: quad epsilon approx 0.033$\"."],[526.4184583333333,"k_live ticks to 2.0."],[527.9744583333334,"k_probe is redrawn as the numbers it depends on change."],[527.9744583333334,"reading becomes \"$k = 3: quad epsilon approx 0.017$\"."],[527.9744583333334,"k_live ticks to 3.0."]]},{"start":540.2654583333333,"say":"Five hash functions gets you under one percent. Seven gets you eight in a thousand, and that is the best this budget can do.","live":null,"does":[[540.7414583333333,"k_probe is redrawn as the numbers it depends on change."],[540.7414583333333,"reading becomes \"$k = 5: quad epsilon approx 0.0094$\"."],[540.7414583333333,"k_live ticks to 5.0."],[544.2244583333334,"k_probe is redrawn as the numbers it depends on change."],[544.2244583333334,"reading becomes \"$k = 7: quad epsilon approx 0.0082$\"."],[544.2244583333334,"k_live ticks to 7.0."]]},{"start":549.4224583333333,"say":"Now watch what happens if you keep going. Ten hash functions is worse than seven. Fourteen is worse than three. The curve turns around and climbs back up.","live":null,"does":[[552.8354583333333,"k_probe is redrawn as the numbers it depends on change."],[552.8354583333333,"reading becomes \"$k = 10: quad epsilon approx 0.0100$\"."],[552.8354583333333,"k_live ticks to 10.0."],[555.8774583333334,"k_probe is redrawn as the numbers it depends on change."],[555.8774583333334,"reading becomes \"$k = 14: quad epsilon approx 0.0184$\"."],[555.8774583333334,"k_live ticks to 14.0."]]},{"start":561.6089583333334,"say":"And the reason is sitting in the formula. Every extra hash function is one more bit a query has to match, which helps you. But it is also one more bit set on every insert, so the exponent grows and the row fills up faster, which hurts you. Past a point the second effect wins.","live":null,"does":[[573.6374583333334,"rate (the \"k n slash m\" part) is emphasized."],[579.7094583333333,"rate (the \"k n slash m\" part) is no longer emphasized."]]},{"start":581.1569583333334,"say":"The minimum is where those two balance, and it lands somewhere clean. The best k is m over n times the natural log of two, which for ten bits an item is six point nine. So you use seven.","live":null,"does":[[586.0444583333333,"optimum is shown on the screen, written out."],[593.3944583333333,"point is shown on the screen, grown."]]},{"start":594.7949583333333,"say":"And there is a check you can run in production. At the optimal k the row ends up with almost exactly half its bits set. Half ones, half zeros. If your filter is much fuller than half, you are running too many hash functions.","live":["rate","reading","optimum","k_plot","head_k","k_curve","k_probe","point"],"does":[[595.3944583333333,"point is hidden from the screen."],[601.1924583333333,"k_probe is indicated — a transient flash."],[609.8179583333333,"head_k is hidden from the screen — left the board."],[609.8179583333333,"k_plot is hidden from the screen — left the board."],[609.8179583333333,"k_curve is hidden from the screen — k_plot left the board."],[609.8179583333333,"k_probe is hidden from the screen — k_plot left the board."],[609.8179583333333,"optimum is hidden from the screen — left the board."],[609.8179583333333,"rate is hidden from the screen — left the board."],[609.8179583333333,"reading is hidden from the screen — left the board."]]},{"start":611.0179583333334,"say":"Which leaves the budget itself. Substitute that optimal k back into the rate, and the whole expression collapses into one number raised to the bits per item. About zero point six two, to the power m over n.","live":[],"does":[[611.0179583333334,"head_budget is shown on the screen, written out."],[617.4274583333333,"collapsed is shown on the screen, written out."],[618.5414583333334,"b_plot is shown on the screen, written out."],[618.5414583333334,"b_curve is shown on the screen, drawn."]]},{"start":625.9684583333333,"say":"So here is the curve an engineer actually wants. Bits per item across the bottom, false positive rate up the side. At eight bits an item, two percent.","live":["collapsed","b_plot","head_budget","b_curve"],"does":[[634.8264583333333,"b_probe is shown on the screen, written out."],[635.8944583333333,"budget_read is shown on the screen, written out."]]},{"start":637.7604583333333,"say":"At ten bits, eight in a thousand. At fourteen, about one in a thousand. At twenty bits an item you are down to seven in a hundred thousand, and you are still using less than three bytes a URL.","live":["collapsed","budget_read","b_plot","head_budget","b_curve","b_probe"],"does":[[638.4454583333334,"b_probe is redrawn as the numbers it depends on change."],[638.4454583333334,"budget_read becomes \"$m slash n = 10: quad epsilon approx 0.0082$\"."],[638.4454583333334,"b_live ticks to 10.0."],[641.9164583333334,"b_probe is redrawn as the numbers it depends on change."],[641.9164583333334,"budget_read becomes \"$m slash n = 14: quad epsilon approx 0.0012$\"."],[641.9164583333334,"b_live ticks to 14.0."],[645.4224583333333,"b_probe is redrawn as the numbers it depends on change."],[645.4224583333333,"budget_read becomes \"$m slash n = 20: quad epsilon approx 0.000067$\"."],[645.4224583333333,"b_live ticks to 20.0."]]},{"start":654.6144583333333,"say":"Every extra bit per item multiplies the error by roughly zero point six two. That is what makes ten the number people quote. It is the first place the curve is properly flat, and it is under one percent.","live":null,"does":[[656.4134583333333,"collapsed (the \"0.62\" part) is emphasized."],[662.2304583333333,"collapsed (the \"0.62\" part) is no longer emphasized."]]},{"start":669.0534583333333,"say":"And look at the shape of it. Going from three bits an item to eight is cheap, and it buys you an enormous amount. After that it gets expensive, because you are paying linearly for an error that is shrinking exponentially.","live":null,"does":[[672.0134583333333,"point_2 is shown on the screen, grown."],[673.1514583333334,"point_3 is shown on the screen, grown."],[674.0134583333333,"point_2 is hidden from the screen."],[675.1514583333334,"point_3 is hidden from the screen."],[683.3739791666667,"b_plot is hidden from the screen — left the board."],[683.3739791666667,"b_curve is hidden from the screen — b_plot left the board."],[683.3739791666667,"b_probe is hidden from the screen — b_plot left the board."],[683.3739791666667,"budget_read is hidden from the screen — left the board."],[683.3739791666667,"collapsed is hidden from the screen — left the board."],[683.3739791666667,"head_budget is hidden from the screen — left the board."]]}]},{"title":"No Deletions, and One Real Use","start":684.4156458333333,"end":857.2411875,"objects":{"arrow_fetch":"an Arrow [green] labelled \"upright(\"definitely new\")\" drawn in crawl (start=(7.8, 2.8), end=(8.9, 1.8))","arrow_in":"an Arrow [gray] drawn in crawl (start=(3.4, 3.2), end=(4.4, 3.2))","arrow_skip":"an Arrow [red] labelled \"upright(\"probably seen\")\" drawn in crawl (start=(7.8, 3.6), end=(8.9, 4.6))","bits":"a Figure (x_range=(-0.5, 16.5), y_range=(-1.6, 1.6), aspect=(17.0, 3.2))","box_fetch":"a Polygon [gray] labelled \"upright(\"fetch\")\" drawn in crawl (vertices=((9.1, 1.1), (11.4, 1.1), (11.4, 2.3), (9.1, 2.3)), filled=False)","box_filter":"a Polygon [blue] labelled \"upright(\"Bloom filter\")\" drawn in crawl (vertices=((4.6, 2.4), (7.6, 2.4), (7.6, 4.0), (4.6, 4.0)), fill_opacity=0.25)","box_link":"a Polygon [gray] labelled \"upright(\"URL found\")\" drawn in crawl (vertices=((0.6, 2.5), (3.2, 2.5), (3.2, 3.9), (0.6, 3.9)), filled=False)","box_skip":"a Polygon [gray] labelled \"upright(\"drop\")\" drawn in crawl (vertices=((9.1, 4.1), (11.4, 4.1), (11.4, 5.3), (9.1, 5.3)), filled=False)","cells":"a Polygon [gray] drawn in bits (vertices=((0.0, 0.0), (1.0, 0.0), (1.0, 1.0), (0.0, 1.0)), filled=False)","cells_10":"a Polygon [gray] drawn in bits (vertices=((9.0, 0.0), (10.0, 0.0), (10.0, 1.0), (9.0, 1.0)), filled=False)","cells_11":"a Polygon [gray] drawn in bits (vertices=((10.0, 0.0), (11.0, 0.0), (11.0, 1.0), (10.0, 1.0)), filled=False)","cells_12":"a Polygon [gray] drawn in bits (vertices=((11.0, 0.0), (12.0, 0.0), (12.0, 1.0), (11.0, 1.0)), filled=False)","cells_13":"a Polygon [gray] drawn in bits (vertices=((12.0, 0.0), (13.0, 0.0), (13.0, 1.0), (12.0, 1.0)), filled=False)","cells_14":"a Polygon [gray] drawn in bits (vertices=((13.0, 0.0), (14.0, 0.0), (14.0, 1.0), (13.0, 1.0)), filled=False)","cells_15":"a Polygon [gray] drawn in bits (vertices=((14.0, 0.0), (15.0, 0.0), (15.0, 1.0), (14.0, 1.0)), filled=False)","cells_16":"a Polygon [gray] drawn in bits (vertices=((15.0, 0.0), (16.0, 0.0), (16.0, 1.0), (15.0, 1.0)), filled=False)","cells_2":"a Polygon [gray] drawn in bits (vertices=((1.0, 0.0), (2.0, 0.0), (2.0, 1.0), (1.0, 1.0)), filled=False)","cells_3":"a Polygon [gray] drawn in bits (vertices=((2.0, 0.0), (3.0, 0.0), (3.0, 1.0), (2.0, 1.0)), filled=False)","cells_4":"a Polygon [gray] drawn in bits (vertices=((3.0, 0.0), (4.0, 0.0), (4.0, 1.0), (3.0, 1.0)), filled=False)","cells_5":"a Polygon [gray] drawn in bits (vertices=((4.0, 0.0), (5.0, 0.0), (5.0, 1.0), (4.0, 1.0)), filled=False)","cells_6":"a Polygon [gray] drawn in bits (vertices=((5.0, 0.0), (6.0, 0.0), (6.0, 1.0), (5.0, 1.0)), filled=False)","cells_7":"a Polygon [gray] drawn in bits (vertices=((6.0, 0.0), (7.0, 0.0), (7.0, 1.0), (6.0, 1.0)), filled=False)","cells_8":"a Polygon [gray] drawn in bits (vertices=((7.0, 0.0), (8.0, 0.0), (8.0, 1.0), (7.0, 1.0)), filled=False)","cells_9":"a Polygon [gray] drawn in bits (vertices=((8.0, 0.0), (9.0, 0.0), (9.0, 1.0), (8.0, 1.0)), filled=False)","clear_chip":"a Math [text] that says \"$upright(\"clear\") thin x: quad 2, thin 7, thin 11$\"","counting":"a Panel that says \"If you truly must delete, replace each bit with a small counter and increment on insert. That is a counting Bloom filter, and four bit counters make the structure four times larger.\"","crawl":"a Figure (x_range=(0.0, 12.0), y_range=(0.0, 6.4), aspect=(12.0, 6.4))","head_crawl":"a Heading that says \"A Crawler That Has Seen a Billion URLs\"","head_delete":"a Heading that says \"Try to Delete Something\"","head_rule":"a Heading that says \"The Rule, Plainly\"","ones":"a Polygon [blue] drawn in bits (vertices=((1.0, 0.0), (2.0, 0.0), (2.0, 1.0), (1.0, 1.0)), fill_opacity=0.75)","ones_2":"a Polygon [blue] drawn in bits (vertices=((2.0, 0.0), (3.0, 0.0), (3.0, 1.0), (2.0, 1.0)), fill_opacity=0.75)","ones_3":"a Polygon [blue] drawn in bits (vertices=((4.0, 0.0), (5.0, 0.0), (5.0, 1.0), (4.0, 1.0)), fill_opacity=0.75)","ones_4":"a Polygon [blue] drawn in bits (vertices=((7.0, 0.0), (8.0, 0.0), (8.0, 1.0), (7.0, 1.0)), fill_opacity=0.75)","ones_5":"a Polygon [blue] drawn in bits (vertices=((9.0, 0.0), (10.0, 0.0), (10.0, 1.0), (9.0, 1.0)), fill_opacity=0.75)","ones_6":"a Polygon [blue] drawn in bits (vertices=((11.0, 0.0), (12.0, 0.0), (12.0, 1.0), (11.0, 1.0)), fill_opacity=0.75)","ones_7":"a Polygon [blue] drawn in bits (vertices=((13.0, 0.0), (14.0, 0.0), (14.0, 1.0), (13.0, 1.0)), fill_opacity=0.75)","ones_8":"a Polygon [blue] drawn in bits (vertices=((14.0, 0.0), (15.0, 0.0), (15.0, 1.0), (14.0, 1.0)), fill_opacity=0.75)","query_chip":"a Math [text] that says \"$upright(\"query\") thin y: quad 4, thin 7, thin 13$\"","rule":"a Panel that says \"A bit set by one item may also be needed by another. Clearing it can turn a present item into a definite absence, and definite absence is the one answer a Bloom filter is never allowed to get wrong.\"","scale_1":"a Math [text] that says \"$m slash n = 10 arrow.r 1.25 thin upright(\"GB\")$\"","scale_2":"a Math [text] that says \"$epsilon approx 0.008$\"","scale_3":"a Math [text] that says \"$8 dot.op 10^6 thin upright(\"pages skipped\")$\"","tags":"a Math [gray] that says \"$0$\" drawn in bits","tags_10":"a Math [gray] that says \"$9$\" drawn in bits","tags_11":"a Math [gray] that says \"$10$\" drawn in bits","tags_12":"a Math [gray] that says \"$11$\" drawn in bits","tags_13":"a Math [gray] that says \"$12$\" drawn in bits","tags_14":"a Math [gray] that says \"$13$\" drawn in bits","tags_15":"a Math [gray] that says \"$14$\" drawn in bits","tags_16":"a Math [gray] that says \"$15$\" drawn in bits","tags_2":"a Math [gray] that says \"$1$\" drawn in bits","tags_3":"a Math [gray] that says \"$2$\" drawn in bits","tags_4":"a Math [gray] that says \"$3$\" drawn in bits","tags_5":"a Math [gray] that says \"$4$\" drawn in bits","tags_6":"a Math [gray] that says \"$5$\" drawn in bits","tags_7":"a Math [gray] that says \"$6$\" drawn in bits","tags_8":"a Math [gray] that says \"$7$\" drawn in bits","tags_9":"a Math [gray] that says \"$8$\" drawn in bits","verdict":"a Math [text] that says \"$upright(\"definitely absent\")$\""},"beats":[{"start":684.4156458333333,"say":"One more property, and it is the one that catches people out. You cannot remove anything from a Bloom filter. Here is why, on the row we already have, with the same three items in it.","live":[],"does":[[684.4156458333333,"head_delete is shown on the screen, written out."],[684.4156458333333,"bits is shown on the screen, written out."],[692.0896458333333,"cells is shown on the screen, faded in."],[692.1296458333334,"cells_2 is shown on the screen, faded in."],[692.1696458333333,"cells_3 is shown on the screen, faded in."],[692.2096458333333,"cells_4 is shown on the screen, faded in."],[692.2496458333333,"cells_5 is shown on the screen, faded in."],[692.2896458333333,"cells_6 is shown on the screen, faded in."],[692.3296458333333,"cells_7 is shown on the screen, faded in."],[692.3696458333333,"cells_8 is shown on the screen, faded in."],[692.4096458333333,"cells_9 is shown on the screen, faded in."],[692.4496458333333,"cells_10 is shown on the screen, faded in."],[692.4896458333333,"cells_11 is shown on the screen, faded in."],[692.5296458333333,"cells_12 is shown on the screen, faded in."],[692.5696458333333,"cells_13 is shown on the screen, faded in."],[692.6096458333333,"cells_14 is shown on the screen, faded in."],[692.6496458333334,"cells_15 is shown on the screen, faded in."],[692.6896458333333,"cells_16 is shown on the screen, faded in."],[694.3306458333333,"ones is shown on the screen, faded in."],[694.3806458333333,"ones_2 is shown on the screen, faded in."],[694.4306458333333,"ones_3 is shown on the screen, faded in."],[694.4806458333334,"ones_4 is shown on the screen, faded in."],[694.5306458333333,"ones_5 is shown on the screen, faded in."],[694.5706458333333,"tags is shown on the screen, written out."],[694.5806458333333,"ones_6 is shown on the screen, faded in."],[694.6006458333333,"tags_2 is shown on the screen, written out."],[694.6306458333333,"ones_7 is shown on the screen, faded in."],[694.6306458333333,"tags_3 is shown on the screen, written out."],[694.6606458333333,"tags_4 is shown on the screen, written out."],[694.6806458333333,"ones_8 is shown on the screen, faded in."],[694.6906458333333,"tags_5 is shown on the screen, written out."],[694.7206458333333,"tags_6 is shown on the screen, written out."],[694.7506458333334,"tags_7 is shown on the screen, written out."],[694.7806458333333,"tags_8 is shown on the screen, written out."],[694.8106458333333,"tags_9 is shown on the screen, written out."],[694.8406458333333,"tags_10 is shown on the screen, written out."],[694.8706458333334,"tags_11 is shown on the screen, written out."],[694.9006458333333,"tags_12 is shown on the screen, written out."],[694.9306458333333,"tags_13 is shown on the screen, written out."],[694.9606458333333,"tags_14 is shown on the screen, written out."],[694.9906458333334,"tags_15 is shown on the screen, written out."],[695.0206458333333,"tags_16 is shown on the screen, written out."]]},{"start":695.9056458333333,"say":"Suppose we want to take the first item back out. Its bits were two, seven, and eleven, so the obvious move is to clear those three.","live":["bits","head_delete","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","ones","ones_2","ones_3","ones_4","ones_5","ones_6","ones_7","ones_8","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16"],"does":[[699.3656458333334,"clear_chip is shown on the screen, written out."],[702.3846458333334,"ones_2 is hidden from the screen."],[702.3846458333334,"ones_4 is hidden from the screen."],[702.3846458333334,"ones_6 is hidden from the screen."]]},{"start":704.0986458333333,"say":"Now ask about the second item, which is still very much in the set. Its bits are four, seven, and thirteen. Four is a one. Thirteen is a one. And seven is now a zero, because we just cleared it.","live":["bits","clear_chip","head_delete","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","ones","ones_3","ones_5","ones_7","ones_8","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16"],"does":[[708.4406458333333,"query_chip is shown on the screen, written out."],[708.8126458333334,"The point (4.5, 0.5) in bits is lit up."],[709.8106458333333,"The point (13.5, 0.5) in bits is lit up."],[714.3156458333333,"The point (7.5, 0.5) in bits is lit up."]]},{"start":717.9226458333333,"say":"So the filter says definitely absent, about an item that is definitely present. That is a false negative, and it is the single answer this structure promised you it would never get wrong.","live":["bits","clear_chip","query_chip","head_delete","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","ones","ones_3","ones_5","ones_7","ones_8","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16"],"does":[[719.8616458333333,"verdict is shown on the screen, written out."]]},{"start":730.0166458333333,"say":"The bit at seven was never the first item's bit to give back. It was shared, and nothing in the row records how many items are leaning on it. So we put it back, and we do not do that again.","live":["bits","clear_chip","query_chip","verdict","head_delete","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","ones","ones_3","ones_5","ones_7","ones_8","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16"],"does":[[734.0446458333333,"bits: retire a lit point (unemphasize_point)."],[734.0446458333333,"bits: retire a lit point (unemphasize_point)."],[734.0446458333333,"bits: retire a lit point (unemphasize_point)."],[739.0026458333333,"ones_4 is shown on the screen, faded in."],[739.0026458333333,"ones_2 is shown on the screen, faded in."],[739.0026458333333,"ones_6 is shown on the screen, faded in."]]},{"start":742.1336458333333,"say":"Which gives us the rule, and it is worth stating plainly.","live":["bits","clear_chip","query_chip","verdict","head_delete","cells","cells_2","cells_3","cells_4","cells_5","cells_6","cells_7","cells_8","cells_9","cells_10","cells_11","cells_12","cells_13","cells_14","cells_15","cells_16","ones","ones_2","ones_3","ones_4","ones_5","ones_6","ones_7","ones_8","tags","tags_2","tags_3","tags_4","tags_5","tags_6","tags_7","tags_8","tags_9","tags_10","tags_11","tags_12","tags_13","tags_14","tags_15","tags_16"],"does":[[746.0001458333334,"bits is hidden from the screen — left the board."],[746.0001458333334,"cells is hidden from the screen — bits left the board."],[746.0001458333334,"cells_2 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_3 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_4 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_5 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_6 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_7 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_8 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_9 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_10 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_11 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_12 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_13 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_14 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_15 is hidden from the screen — bits left the board."],[746.0001458333334,"cells_16 is hidden from the screen — bits left the board."],[746.0001458333334,"ones is hidden from the screen — bits left the board."],[746.0001458333334,"ones_2 is hidden from the screen — bits left the board."],[746.0001458333334,"ones_3 is hidden from the screen — bits left the board."],[746.0001458333334,"ones_4 is hidden from the screen — bits left the board."],[746.0001458333334,"ones_5 is hidden from the screen — bits left the board."],[746.0001458333334,"ones_6 is hidden from the screen — bits left the board."],[746.0001458333334,"ones_7 is hidden from the screen — bits left the board."],[746.0001458333334,"ones_8 is hidden from the screen — bits left the board."],[746.0001458333334,"tags is hidden from the screen — bits left the board."],[746.0001458333334,"tags_2 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_3 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_4 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_5 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_6 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_7 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_8 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_9 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_10 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_11 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_12 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_13 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_14 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_15 is hidden from the screen — bits left the board."],[746.0001458333334,"tags_16 is hidden from the screen — bits left the board."],[746.0001458333334,"clear_chip is hidden from the screen — left the board."],[746.0001458333334,"head_delete is hidden from the screen — left the board."],[746.0001458333334,"query_chip is hidden from the screen — left the board."],[746.0001458333334,"verdict is hidden from the screen — left the board."]]},{"start":747.2001458333333,"say":"A Bloom filter supports insert, and it supports query, and it does not support delete. If the things you are tracking ever go away again, this is the wrong structure for them.","live":[],"does":[[747.2001458333333,"head_rule is shown on the screen, written out."],[748.7096458333333,"rule is shown on the screen, written out."],[752.2506458333334,"rule (the \"Clearing it\" part) is emphasized."],[756.5926458333333,"rule (the \"Clearing it\" part) is no longer emphasized."]]},{"start":758.7831458333333,"say":"There is a variant that fixes it. Replace each bit with a small counter, increment on insert, decrement on delete, and now the shared position knows that two items are leaning on it. You pay for that. Four bit counters make the whole thing four times bigger, which was most of the reason you reached for a Bloom filter in the first place.","live":["rule","head_rule"],"does":[[759.6536458333333,"counting is shown on the screen, written out."],[777.8581458333333,"counting is hidden from the screen — left the board."],[777.8581458333333,"head_rule is hidden from the screen — left the board."],[777.8581458333333,"rule is hidden from the screen — left the board."]]},{"start":779.0581458333334,"say":"So where does this earn its keep? A crawler is the clean case. You pull a URL off the frontier, and before you spend a network round trip on it, you ask whether you have fetched it already.","live":[],"does":[[779.0581458333334,"head_crawl is shown on the screen, written out."],[779.0581458333334,"crawl is shown on the screen, written out."],[784.5266458333333,"box_link is shown on the screen, written out."],[788.7176458333333,"arrow_in is shown on the screen, drawn."],[788.7176458333333,"box_filter is shown on the screen, written out."]]},{"start":791.2216458333332,"say":"The filter answers in the two ways we know. Definitely new, and you fetch it. Probably seen, and you drop it on the floor.","live":["crawl","head_crawl","box_link","arrow_in","box_filter"],"does":[[794.3446458333333,"arrow_fetch is shown on the screen, drawn."],[795.9006458333333,"box_fetch is shown on the screen, written out."],[797.0496458333333,"arrow_skip is shown on the screen, drawn."],[798.4666458333332,"box_skip is shown on the screen, written out."]]},{"start":800.4011458333333,"say":"A billion URLs at ten bits each is a gigabyte and a quarter, which fits in memory on any machine you would run a crawler on. The exact set was sixty four gigabytes, and a disk seek for every link on every page.","live":["crawl","head_crawl","box_link","arrow_in","box_filter","arrow_fetch","box_fetch","arrow_skip","box_skip"],"does":[[804.1286458333333,"crawl moves to a new place on the board."],[804.1286458333333,"scale_1 is shown on the screen, written out."]]},{"start":814.8176458333332,"say":"At seven hash functions the rate is eight in a thousand, so about eight million URLs out of a billion get reported as seen when they never were, and never get fetched. On a web that size, that is a rounding error in your coverage.","live":["scale_1","crawl","head_crawl","box_link","arrow_in","box_filter","arrow_fetch","box_fetch","arrow_skip","box_skip"],"does":[[817.2676458333333,"scale_2 is shown on the screen, written out."],[823.4786458333333,"scale_3 is shown on the screen, written out."]]},{"start":828.9436458333333,"say":"And notice which way the errors fall. What cannot happen is a page you have already fetched coming back as new. That is the one that puts a crawler in a loop, and it is precisely the answer the filter is exact about.","live":["scale_1","scale_2","scale_3","crawl","head_crawl","box_link","arrow_in","box_filter","arrow_fetch","box_fetch","arrow_skip","box_skip"],"does":[[832.1136458333333,"arrow_fetch is indicated — a transient flash."],[838.1506458333333,"box_fetch is indicated — a transient flash."]]},{"start":842.8256458333333,"say":"So that is the whole structure. Bits instead of items. k hash functions instead of a comparison. A false positive rate you can compute before you ship. And no deletions, ever.","live":null,"does":[[856.1995208333333,"crawl is hidden from the screen — left the board."],[856.1995208333333,"box_link is hidden from the screen — crawl left the board."],[856.1995208333333,"arrow_in is hidden from the screen — crawl left the board."],[856.1995208333333,"box_filter is hidden from the screen — crawl left the board."],[856.1995208333333,"arrow_fetch is hidden from the screen — crawl left the board."],[856.1995208333333,"box_fetch is hidden from the screen — crawl left the board."],[856.1995208333333,"arrow_skip is hidden from the screen — crawl left the board."],[856.1995208333333,"box_skip is hidden from the screen — crawl left the board."],[856.1995208333333,"head_crawl is hidden from the screen — left the board."],[856.1995208333333,"scale_1 is hidden from the screen — left the board."],[856.1995208333333,"scale_2 is hidden from the screen — left the board."],[856.1995208333333,"scale_3 is hidden from the screen — left the board."]]}]}]},"durationSeconds":857,"chapters":[{"title":"A Set You Cannot Afford to Keep","startSeconds":0,"narration":"Here is a problem that turns up in almost every large system. Items arrive one at a time, and for each one you have to answer a single question. Have I seen this one before? A crawler asks it of every link it discovers, and a cache asks it of every key. So here is the question this lecture answers. Have I seen this item before, when the set of items I have seen is far too large to keep? The second half of that sentence is where all of the difficulty lives. The obvious answer is to keep the set. Every item you have seen goes into a hash set, and every new one gets looked up. Across the bottom here is n, the number of items, counted in millions. Up the side is memory, in gigabytes. URLs are not small. Sixty four bytes each is a kind estimate, and this red line is what that costs you. A billion of them is sixty four gigabytes, and it is a straight line, because an exact set charges you for every item, forever. The dashed line is the memory the machine actually has. Eight gigabytes. The red line crosses it here, at about a hundred and twenty five million URLs, and past that point the structure does not fit and the process dies. The blue line is where we are going. The same billion URLs, about ten bits each, and the whole structure is a gigabyte and a quarter. Fifty times smaller, and it never stores a single URL. You do not get that for free. What you give up is certainty, in one direction only. A Bloom filter answers two ways: definitely not in the set, or probably in the set. The second of those is sometimes wrong. Two things to hold on to before we build it. It never stores the items, so you can never ask it what is in there. And the mistakes only ever go one way. It will never tell you an item is absent when you did in fact insert it."},{"title":"Setting the Bits","startSeconds":125.44514583333334,"narration":"So we are not going to store the items. We are going to store a fixed row of bits, and nothing else. Here is one: sixteen bits, numbered zero to fifteen, and every single one of them is zero. Call the length m. In a real filter m is not sixteen, it is a few billion. Sixteen is a size you can see. And notice what is missing here. There is no list of items anywhere in this picture, no copy of anything that was ever inserted. There are only bits. So take an item. Call it x. We hash it, not once, but k times, with k independent hash functions. Here k is three, and the first of them hands back position two. The second gives seven, and the third gives eleven. Three positions, three bits, and we set all three of them to one. That is the whole of the insert operation. Notice what did not happen there. We did not store x. We did not store a hash of x, or a pointer to x, or how long it was. Three bits went from zero to one, and the item itself is gone. Insert a second item. Its three hashes come out four, seven, and thirteen, so we set those. And look at position seven. It was already a one, set by the first item, and it stays a one. The row has no idea that two different items are now leaning on that single bit. Hold on to that. A third item, and its positions are one, nine, and fourteen. Three items are in the filter now, and eight of the sixteen bits are ones. That is the entire state. So now the other half of the job. How do you ask a question? To test an item you hash it exactly the same way, and you look at the k bits it names. Here is the second item again. Four, seven, thirteen. All three of them are ones. So the filter says: probably present. And it is right, we did insert that one. But look at what it actually checked. It never compared the item to anything at all. It read three bits. Now an item that was never inserted. Its positions come out zero, nine, and fourteen. The bit at nine is a one, the bit at fourteen is a one, and the bit at zero is not. And that settles it. If this item had ever been inserted, every one of its bits would have been set at that moment, and bits in this row are never cleared. A single zero anywhere is proof of absence. Definitely not in the set. That answer is exact, and it is the one guarantee the structure gives you. It is worth more than it sounds. Most of the time the thing you are asking about really is new, and you get told so for three bit lookups and no disk at all."},{"title":"The False Positive","startSeconds":305.84783333333337,"narration":"Here is that same row, with those same three items in it, and now I am going to ask about an item that nobody ever inserted. Call it v. Its three positions come out two, four, and nine. Two is a one. The first item set it. Four is a one. The second item set that. Nine is a one, and the third item set that. So all k bits are ones, and the filter says: probably present. And it is wrong. No item set that pattern. Three different items each happened to set one of its bits, and between them they covered it. That is a false positive. It is not a bug, and there is no clever fix for it. It is what you bought when you decided to store bits instead of items. The bits do not remember who set them. What you can do is work out how often it happens, before you ship anything, and then choose how often you are willing to let it happen. So let us count it. Fix one bit in the row and follow it. Call P nought the chance that this bit is still a zero, and P one the chance that something has set it. A single hash function picks a position uniformly out of m, so it misses our bit with probability one minus one over m. Inserting one item runs k hashes. Inserting n items runs k times n of them, and if the hash functions are independent our bit survives every one of them, with probability one minus one over m, all raised to the power k n. That form is awkward to think with, so use the standard approximation. When m is large it is very close to e to the minus k n over m. Same number, and far easier to see what it does. Turn it around. The chance that a given bit has been set is one minus that. And this curve is exactly that number, for our sixteen bit row with three hash functions, as n climbs. With three items in, the formula says about forty three percent of the bits are ones. We counted eight out of sixteen, which is fifty. With only sixteen bits that is the sort of luck you get. Now push n up to twelve. Nearly ninety percent of the row is ones, and at that point almost any pattern you ask about is already covered by something. The false positive rate climbs with n, and this is the reason why. So now the query itself. An item that was never inserted names k bits. Each one is a one with probability P one, and if we treat those as independent, all k of them are ones with probability P one to the k. There it is. The false positive rate, epsilon, is one minus e to the minus k n over m, all raised to the k. Read the exponent as the total number of bits you have set, k of them for each of n items, divided by the room you gave them to spread out in. Read the rest of it the way an engineer reads it. n is handed to you by the problem. m is what you are willing to pay for. And k is the one you actually get to choose, and it is not at all obvious which way to push it."},{"title":"Choosing the Hash Functions","startSeconds":501.55745833333333,"narration":"So, choosing k. Fix the budget at ten bits an item, which is the number you will see quoted everywhere, and the rate depends on k and nothing else. Here it is, plotted against the number of hash functions. With one hash function, each item sets a single bit, and the rate is about nine and a half percent. Nearly one query in ten comes back wrong. Two hash functions: three point three percent. Three: one point seven. Every extra hash function is another bit that has to be a one by accident, and that gets rapidly less likely. Five hash functions gets you under one percent. Seven gets you eight in a thousand, and that is the best this budget can do. Now watch what happens if you keep going. Ten hash functions is worse than seven. Fourteen is worse than three. The curve turns around and climbs back up. And the reason is sitting in the formula. Every extra hash function is one more bit a query has to match, which helps you. But it is also one more bit set on every insert, so the exponent grows and the row fills up faster, which hurts you. Past a point the second effect wins. The minimum is where those two balance, and it lands somewhere clean. The best k is m over n times the natural log of two, which for ten bits an item is six point nine. So you use seven. And there is a check you can run in production. At the optimal k the row ends up with almost exactly half its bits set. Half ones, half zeros. If your filter is much fuller than half, you are running too many hash functions. Which leaves the budget itself. Substitute that optimal k back into the rate, and the whole expression collapses into one number raised to the bits per item. About zero point six two, to the power m over n. So here is the curve an engineer actually wants. Bits per item across the bottom, false positive rate up the side. At eight bits an item, two percent. At ten bits, eight in a thousand. At fourteen, about one in a thousand. At twenty bits an item you are down to seven in a hundred thousand, and you are still using less than three bytes a URL. Every extra bit per item multiplies the error by roughly zero point six two. That is what makes ten the number people quote. It is the first place the curve is properly flat, and it is under one percent. And look at the shape of it. Going from three bits an item to eight is cheap, and it buys you an enormous amount. After that it gets expensive, because you are paying linearly for an error that is shrinking exponentially."},{"title":"No Deletions, and One Real Use","startSeconds":684.4156458333333,"narration":"One more property, and it is the one that catches people out. You cannot remove anything from a Bloom filter. Here is why, on the row we already have, with the same three items in it. Suppose we want to take the first item back out. Its bits were two, seven, and eleven, so the obvious move is to clear those three. Now ask about the second item, which is still very much in the set. Its bits are four, seven, and thirteen. Four is a one. Thirteen is a one. And seven is now a zero, because we just cleared it. So the filter says definitely absent, about an item that is definitely present. That is a false negative, and it is the single answer this structure promised you it would never get wrong. The bit at seven was never the first item's bit to give back. It was shared, and nothing in the row records how many items are leaning on it. So we put it back, and we do not do that again. Which gives us the rule, and it is worth stating plainly. A Bloom filter supports insert, and it supports query, and it does not support delete. If the things you are tracking ever go away again, this is the wrong structure for them. There is a variant that fixes it. Replace each bit with a small counter, increment on insert, decrement on delete, and now the shared position knows that two items are leaning on it. You pay for that. Four bit counters make the whole thing four times bigger, which was most of the reason you reached for a Bloom filter in the first place. So where does this earn its keep? A crawler is the clean case. You pull a URL off the frontier, and before you spend a network round trip on it, you ask whether you have fetched it already. The filter answers in the two ways we know. Definitely new, and you fetch it. Probably seen, and you drop it on the floor. A billion URLs at ten bits each is a gigabyte and a quarter, which fits in memory on any machine you would run a crawler on. The exact set was sixty four gigabytes, and a disk seek for every link on every page. At seven hash functions the rate is eight in a thousand, so about eight million URLs out of a billion get reported as seen when they never were, and never get fetched. On a web that size, that is a rounding error in your coverage. And notice which way the errors fall. What cannot happen is a page you have already fetched coming back as new. That is the one that puts a crawler in a loop, and it is precisely the answer the filter is exact about. So that is the whole structure. Bits instead of items. k hash functions instead of a comparison. A false positive rate you can compute before you ship. And no deletions, ever."}]}}