{"version":1,"lectureId":"01M14TXDQBTMXE7F1ES780ET6G","attempt":2,"publication":{"slug":"entropy-as-twenty-questions","title":"From Twenty Questions to Entropy and Huffman Compression","subject":"computer-science","summary":"A concrete introduction to information theory through binary questions. Eight equally likely outcomes establish the bit, then an unequal distribution shows why likely symbols deserve shorter descriptions. Repeated halving leads to self-information and the entropy formula. The same probabilities are used to build a Huffman tree one rarest-pair merge at a time, producing an average length of 2.60 bits per symbol against an entropy of about 2.522. The lecture closes with the binary tree capacity argument behind the entropy lower bound and explains why predictable English text compresses while independent random bytes generally do not.","metaDescription":"Derive entropy from twenty questions, build a Huffman code, compare its length with the limit, and explain when files can compress.","transcript":"Suppose I secretly choose one of these eight letters. You may ask only questions whose answer is yes or no. Your job is to identify my letter with certainty. How many answers do you need? Because the letters are equally likely, the sensible first question cuts the possibilities into equal halves. Ask whether the letter lies in A through D. One answer leaves the upper four; the other leaves the lower four. The first answer has taken us from eight candidates to four. It has not named the symbol, but it has removed exactly half of the uncertainty. Ask another balanced question inside the surviving half. Four candidates become two. The same yes-or-no answer has again divided the remaining possibilities by two. A third question separates the final pair. Eight leaves now stand at the ends of the tree, and every letter is reached by exactly three answers. The count is not an accident. Three binary answers can form eight distinct answer strings, because two choices made three times give two cubed. Turn that relation around. The number of binary questions needed to choose among eight equal possibilities is log base two of eight, which is three. We give one balanced binary answer a unit of information and call that unit a bit. A bit is not merely a zero stored in memory. It is the distinction between two alternatives that were both still possible. With q binary answers there are two to the q possible answer strings. Therefore identifying one of N equally likely outcomes requires log base two of N bits. Next we will make the letters unequally likely. That one change will lead us first to entropy, the ideal average question count, and then to Huffman coding, a way to build an actual binary description close to that ideal. Keep the same eight symbols, but change their likelihoods. A now occurs forty percent of the time. B occurs twenty percent. C and D each occur ten percent, and the final four symbols each occur five percent. A alone occupies forty percent of the probability bar. It is twice as likely as B, and four times as likely as either C or D. C and D together occupy another twenty percent. They are less common individually, but they still matter much more often than any one of the four symbols at the narrow end. The final four slices are equal and rare. Each accounts for only one twentieth of the messages. If we keep the old balanced tree, every outcome still costs three questions. That treats a five-percent event exactly like the event that appears forty percent of the time. Instead, ask the most useful question first: is the symbol A? When the answer is yes, which happens forty percent of the time, the search ends after one bit. If it was not A, the next question can separate B. B then takes two answers. The commonest sixty percent of all symbols have received the two shortest paths. The price is that C and D now require four questions. That sounds worse than the old three, but each of those symbols appears only one time in ten. The four rarest outcomes take five questions each. Long answers are not free, but we pay for them on only twenty percent of the symbols. Now average the question counts using their probabilities. A contributes point four times one. B contributes point two times two. C and D together contribute point two times four, and the rare group contributes point two times five. The result is two point six questions per symbol. Some individual outcomes became more expensive, but the outcomes we see most often became much cheaper. The equal-depth plan cost exactly three questions. Exploiting likelihood saves point four of a question per symbol on average. Over a million symbols, that becomes roughly four hundred thousand fewer binary answers. For eight equal outcomes, three halvings isolate one outcome because one eighth equals one over two cubed. Now replace one eighth by a general probability p. Imagine a very large collection of equally detailed possibilities. An event with probability p occupies the fraction p of them. To isolate that fraction by balanced binary questions, we keep halving. After q balanced questions, one answer path identifies about one part in two to the q. To match an event occupying fraction p, two to the q must be about one over p. Solve for q. The ideal question count is log base two of one over p. Equivalently, it is minus log base two of p. This quantity measures surprise. A likely event has a large p and a small information value. A rare event has a small p and needs a longer answer path to distinguish it. For A, p is point four, so the ideal length is about one point three two two bits. For a five-percent symbol, it is about four point three two two bits. These are not yet legal single-symbol code lengths. A single binary question cannot be split into a fraction. But across a long stream, common and rare symbols can share a description strategy, and a fractional average becomes meaningful. To find the ideal average, weight each symbol's question count by the probability of seeing that symbol, then add. This weighted average is the quantity we call entropy. Now calculate it for our distribution. A contributes about point five two nine bits per source symbol. That is its ideal length multiplied by the forty-percent frequency with which we pay it. B contributes about point four six four. C and D together contribute about point six six four. The four five-percent symbols each carry more surprise, and together they contribute about point eight six four bits. Adding every group gives an entropy of about two point five two two bits per symbol. Here is another way to read the same number. In a long block of n symbols, the probability pattern concentrates most messages among roughly two to the n H typical sequences. Distinguishing that many likely messages needs about the log of their count, which is n H binary answers. Entropy is therefore not an arbitrary formula. It is the question count forced by the number of likely messages. Huffman coding turns the probability list into a binary tree. Begin with one leaf for every symbol. The only repeated instruction is to merge the two smallest weights. The first two rarest leaves are E and F, each with weight five. Join them under a new node of weight ten. G and H are the next equal pair. Merge their five-percent weights into a second node of weight ten. C and D each have weight ten. Merge those two leaves into a node of weight twenty. The two ten-weight branches made from E through H are now the smallest pair. Their merge produces another weight twenty. Three weights of twenty are available. Ties may be resolved consistently in more than one way. Merge these two compound branches to make forty. B has weight twenty and the compound branch has weight forty. They are the two smallest remaining items, so their parent has weight sixty. Finally merge A, with weight forty, and the remaining weight sixty. Their root has weight one hundred, and the Huffman tree is complete. Assign zero to the upper branch and one to the lower branch at each split. A codeword is the sequence encountered while walking from the root to a leaf. A is reached immediately by code zero, so the most likely symbol costs only one bit. B follows one then zero, so it costs two bits. C and D sit deeper, with codes one one zero zero and one one zero one. Each requires four bits. The four rarest symbols lie one level deeper still. Their codes share the prefix one one one, then use two more bits to select the leaf. No codeword is the beginning of another codeword. That prefix property lets a decoder read the stream from left to right and know exactly when each symbol ends. Compute the average length. A pays one bit forty percent of the time. B pays two bits twenty percent of the time. C and D pay four, and the final four symbols pay five. The Huffman code therefore uses two point six bits per symbol on average. The merging rule has recovered the likelihood-aware question plan, and it has guaranteed that no other binary prefix code has a smaller average length for these individual symbols. Put the two numbers on one scale. Entropy for our source is about two point five two two bits per symbol. The Huffman code uses two point six. The gap is only point zero seven eight of a bit per symbol. Huffman coding is not merely better than the fixed three-bit code. It is very close to the information-theoretic target. Two questions remain. Why can no lossless prefix code move the green point to the left of entropy? And if the bound is unbeatable, why is Huffman allowed to sit slightly above it? A binary prefix code is a set of nonoverlapping leaves in a binary tree. A codeword of length l occupies one part in two to the l of the tree's capacity. Add those occupied fractions over all symbols. They cannot exceed one. This is the tree-capacity inequality, often called Kraft's inequality. Now compare the actual average length L with entropy H. Their difference can be written as this weighted logarithmic ratio. The log-sum inequality says that weighted ratio is at least minus the log of the tree capacity. Since the capacity is at most one, that quantity is nonnegative. Therefore every binary prefix code has average length at least H. Shorter paths for some outcomes consume more tree capacity and force compensating longer paths elsewhere. The average cannot cross the entropy bound. Entropy supplies ideal lengths such as one point three two two bits. Real codewords have whole-number lengths. Huffman chooses those integers optimally, and its average is always less than one bit above entropy for a memoryless symbol source. When probabilities are exact powers of one half, the ideal lengths are integers and Huffman can meet entropy exactly. Otherwise it rounds through the tree, as our small point zero seven eight gap demonstrates. There is also a way to make the rounding cost per symbol smaller. Encode blocks of symbols together. One possible extra bit is then shared by the whole block rather than charged to every symbol. Now return to files. Typical English text is structured. Spaces and common letters occur more often than rare punctuation, so even the one-byte distribution is uneven. English also has dependencies across positions. After the letters q and u, some next letters are far more plausible than others. Words, phrases, indentation, and repeated substrings create further predictability. A compressor turns that predictability into short descriptions. Huffman coding exploits a probability distribution. More advanced compressors also exploit context, repeated strings, and predictions from earlier data. Independent random bytes are different. Each of the two hundred fifty-six byte values has probability one over two hundred fifty-six, and the next byte is not predicted by the bytes before it. There is no statistical redundancy to exploit. The entropy is eight bits per byte, exactly the size already used by an ordinary byte. There are two hundred fifty-six possible byte values, which is two to the eighth. If they are equally likely, identifying one requires eight bits. For an n-byte file there are two to the eight n possible inputs. There are not enough shorter binary strings to give every input a unique shorter lossless description. Some individual random-looking file may shrink by chance, but most will not, and compressor metadata can make them larger. Already compressed or encrypted files often behave similarly because their byte patterns have been deliberately made difficult to predict. The whole story now fits into four steps. First, a bit is one resolved binary choice. Second, repeated halving says an event of probability p carries about log base two of one over p bits. Averaging that surprise gives entropy. Third, Huffman coding repeatedly merges rare events so likely symbols end near the root. Its integer code lengths approach the entropy limit without ever beating that lower bound. Finally, compression is possible when data contains predictable structure. English supplies redundancy and context. Independent random bytes do not. Entropy measures that difference, and coding turns the available predictability into fewer stored bits.","watch":{"version":1,"scenes":[{"title":"Questions Are Bits","start":0,"end":119.25329166666668,"objects":{"binary_note":"a Math [text] that says \"$q thin upright(\"binary questions\") arrow.r 2^q thin upright(\"possible answers\")$\"","bit_definition":"a Panel that says \"One bit is the information supplied by one answer to a binary question, when either answer remains possible.\"","bit_line":"a Math [text] that says \"$log_2 8 = 3 thin upright(\"bits\")$\"","count_line":"a Math [text] that says \"$8 arrow.r 4 arrow.r 2 arrow.r 1$\"","destination":"a Text [text] that says \"Our destination is a number called entropy and a practical construction called Huffman coding. Both answer the same question: how short can the average description of a symbol be?\"","first_edges":"a Line [yellow] drawn in tree (start=(0.45, 2.0), end=(1.8, 3.0))","first_edges_2":"a Line [yellow] drawn in tree (start=(0.45, 2.0), end=(1.8, 1.0))","heading":"a Heading that says \"One Answer, One Bit\"","heading_2":"a Heading that says \"Where the Questions Lead\"","leaves":"a Point [blue] labelled \"A\" drawn in tree (location=(5.5, 3.75))","leaves_2":"a Point [blue] labelled \"B\" drawn in tree (location=(5.5, 3.25))","leaves_3":"a Point [blue] labelled \"C\" drawn in tree (location=(5.5, 2.75))","leaves_4":"a Point [blue] labelled \"D\" drawn in tree (location=(5.5, 2.25))","leaves_5":"a Point [blue] labelled \"E\" drawn in tree (location=(5.5, 1.75))","leaves_6":"a Point [blue] labelled \"F\" drawn in tree (location=(5.5, 1.25))","leaves_7":"a Point [blue] labelled \"G\" drawn in tree (location=(5.5, 0.75))","leaves_8":"a Point [blue] labelled \"H\" drawn in tree (location=(5.5, 0.25))","level_one":"a Point [yellow] drawn in tree (location=(1.8, 3.0))","level_one_2":"a Point [yellow] drawn in tree (location=(1.8, 1.0))","level_two":"a Point [green] drawn in tree (location=(3.15, 3.5))","level_two_2":"a Point [green] drawn in tree (location=(3.15, 2.5))","level_two_3":"a Point [green] drawn in tree (location=(3.15, 1.5))","level_two_4":"a Point [green] drawn in tree (location=(3.15, 0.5))","power_line":"a Math [text] that says \"$8 = 2^3$\"","question":"a Panel that says \"I secretly choose one symbol from $A,B,C,D,E,F,G,H$. How many yes-or-no questions are needed to identify it?\"","root":"a Point [red] drawn in tree (location=(0.45, 2.0))","second_edges":"a Line [green] drawn in tree (start=(1.8, 3.0), end=(3.15, 3.5))","second_edges_2":"a Line [green] drawn in tree (start=(1.8, 3.0), end=(3.15, 2.5))","second_edges_3":"a Line [green] drawn in tree (start=(1.8, 1.0), end=(3.15, 1.5))","second_edges_4":"a Line [green] drawn in tree (start=(1.8, 1.0), end=(3.15, 0.5))","third_edges":"a Line [blue] drawn in tree (start=(3.15, 3.5), end=(5.5, 3.75))","third_edges_2":"a Line [blue] drawn in tree (start=(3.15, 3.5), end=(5.5, 3.25))","third_edges_3":"a Line [blue] drawn in tree (start=(3.15, 2.5), end=(5.5, 2.75))","third_edges_4":"a Line [blue] drawn in tree (start=(3.15, 2.5), end=(5.5, 2.25))","third_edges_5":"a Line [blue] drawn in tree (start=(3.15, 1.5), end=(5.5, 1.75))","third_edges_6":"a Line [blue] drawn in tree (start=(3.15, 1.5), end=(5.5, 1.25))","third_edges_7":"a Line [blue] drawn in tree (start=(3.15, 0.5), end=(5.5, 0.75))","third_edges_8":"a Line [blue] drawn in tree (start=(3.15, 0.5), end=(5.5, 0.25))","tree":"a Figure (x_range=(0.0, 6.5), y_range=(0.0, 4.0), aspect=(6.5, 4.0))"},"beats":[{"start":0,"say":"Suppose I secretly choose one of these eight letters. You may ask only questions whose answer is yes or no. Your job is to identify my letter with certainty. How many answers do you need?","live":[],"does":[[0,"question is shown on the screen, written out."],[11.738,"question moves to a new place on the board."]]},{"start":12.338,"say":"Because the letters are equally likely, the sensible first question cuts the possibilities into equal halves. Ask whether the letter lies in A through D. One answer leaves the upper four; the other leaves the lower four.","live":["question"],"does":[[12.338,"tree is shown on the screen, written out."],[12.338,"root is shown on the screen, written out."],[18.141999999999996,"level_one is shown on the screen, written out."],[18.141999999999996,"level_one_2 is shown on the screen, written out."],[18.141999999999996,"first_edges is shown on the screen, written out."],[18.141999999999996,"first_edges_2 is shown on the screen, written out."]]},{"start":26.6485,"say":"The first answer has taken us from eight candidates to four. It has not named the symbol, but it has removed exactly half of the uncertainty.","live":["question","tree","root","level_one","level_one_2","first_edges","first_edges_2"],"does":[[28.621999999999996,"tree moves to a new place on the board."],[28.621999999999996,"count_line is shown on the screen, written out."],[29.517,"count_line (the \"4\" part) is indicated — a transient flash."]]},{"start":35.7245,"say":"Ask another balanced question inside the surviving half. Four candidates become two. The same yes-or-no answer has again divided the remaining possibilities by two.","live":["question","tree","count_line","root","level_one","level_one_2","first_edges","first_edges_2"],"does":[[40.717,"level_two is shown on the screen, written out."],[40.717,"level_two_2 is shown on the screen, written out."],[40.717,"level_two_3 is shown on the screen, written out."],[40.717,"level_two_4 is shown on the screen, written out."],[40.717,"second_edges is shown on the screen, written out."],[40.717,"second_edges_2 is shown on the screen, written out."],[40.717,"second_edges_3 is shown on the screen, written out."],[40.717,"second_edges_4 is shown on the screen, written out."],[45.976,"count_line (the \"2\" part) is indicated — a transient flash."]]},{"start":47.4465,"say":"A third question separates the final pair. Eight leaves now stand at the ends of the tree, and every letter is reached by exactly three answers.","live":["question","tree","count_line","root","level_one","level_one_2","first_edges","first_edges_2","level_two","level_two_2","level_two_3","level_two_4","second_edges","second_edges_2","second_edges_3","second_edges_4"],"does":[[49.351,"count_line (the \"1\" part) is indicated — a transient flash."],[50.906,"leaves is shown on the screen, written out."],[50.906,"leaves_2 is shown on the screen, written out."],[50.906,"leaves_3 is shown on the screen, written out."],[50.906,"leaves_4 is shown on the screen, written out."],[50.906,"leaves_5 is shown on the screen, written out."],[50.906,"leaves_6 is shown on the screen, written out."],[50.906,"leaves_7 is shown on the screen, written out."],[50.906,"leaves_8 is shown on the screen, written out."],[55.515,"third_edges is shown on the screen, written out."],[55.515,"third_edges_2 is shown on the screen, written out."],[55.515,"third_edges_3 is shown on the screen, written out."],[55.515,"third_edges_4 is shown on the screen, written out."],[55.515,"third_edges_5 is shown on the screen, written out."],[55.515,"third_edges_6 is shown on the screen, written out."],[55.515,"third_edges_7 is shown on the screen, written out."],[55.515,"third_edges_8 is shown on the screen, written out."]]},{"start":57.497,"say":"The count is not an accident. Three binary answers can form eight distinct answer strings, because two choices made three times give two cubed.","live":["question","tree","count_line","root","level_one","level_one_2","first_edges","first_edges_2","level_two","level_two_2","level_two_3","level_two_4","second_edges","second_edges_2","second_edges_3","second_edges_4","leaves","leaves_2","leaves_3","leaves_4","leaves_5","leaves_6","leaves_7","leaves_8","third_edges","third_edges_2","third_edges_3","third_edges_4","third_edges_5","third_edges_6","third_edges_7","third_edges_8"],"does":[[57.497,"power_line is shown on the screen, written out."],[65.868,"power_line (the \"2^3\" part) is indicated — a transient flash."]]},{"start":67.536,"say":"Turn that relation around. The number of binary questions needed to choose among eight equal possibilities is log base two of eight, which is three.","live":["question","tree","count_line","power_line","root","level_one","level_one_2","first_edges","first_edges_2","level_two","level_two_2","level_two_3","level_two_4","second_edges","second_edges_2","second_edges_3","second_edges_4","leaves","leaves_2","leaves_3","leaves_4","leaves_5","leaves_6","leaves_7","leaves_8","third_edges","third_edges_2","third_edges_3","third_edges_4","third_edges_5","third_edges_6","third_edges_7","third_edges_8"],"does":[[74.94300000000001,"bit_line is shown on the screen, written out."],[76.77799999999999,"bit_line (the \"3\" part) is indicated — a transient flash."],[77.69500000000001,"bit_line is hidden from the screen — left the board."],[77.69500000000001,"count_line is hidden from the screen — left the board."],[77.69500000000001,"power_line is hidden from the screen — left the board."],[77.69500000000001,"question is hidden from the screen — left the board."],[77.69500000000001,"tree is hidden from the screen — left the board."],[77.69500000000001,"root is hidden from the screen — tree left the board."],[77.69500000000001,"level_one is hidden from the screen — tree left the board."],[77.69500000000001,"level_one_2 is hidden from the screen — tree left the board."],[77.69500000000001,"first_edges is hidden from the screen — tree left the board."],[77.69500000000001,"first_edges_2 is hidden from the screen — tree left the board."],[77.69500000000001,"level_two is hidden from the screen — tree left the board."],[77.69500000000001,"level_two_2 is hidden from the screen — tree left the board."],[77.69500000000001,"level_two_3 is hidden from the screen — tree left the board."],[77.69500000000001,"level_two_4 is hidden from the screen — tree left the board."],[77.69500000000001,"second_edges is hidden from the screen — tree left the board."],[77.69500000000001,"second_edges_2 is hidden from the screen — tree left the board."],[77.69500000000001,"second_edges_3 is hidden from the screen — tree left the board."],[77.69500000000001,"second_edges_4 is hidden from the screen — tree left the board."],[77.69500000000001,"leaves is hidden from the screen — tree left the board."],[77.69500000000001,"leaves_2 is hidden from the screen — tree left the board."],[77.69500000000001,"leaves_3 is hidden from the screen — tree left the board."],[77.69500000000001,"leaves_4 is hidden from the screen — tree left the board."],[77.69500000000001,"leaves_5 is hidden from the screen — tree left the board."],[77.69500000000001,"leaves_6 is hidden from the screen — tree left the board."],[77.69500000000001,"leaves_7 is hidden from the screen — tree left the board."],[77.69500000000001,"leaves_8 is hidden from the screen — tree left the board."],[77.69500000000001,"third_edges is hidden from the screen — tree left the board."],[77.69500000000001,"third_edges_2 is hidden from the screen — tree left the board."],[77.69500000000001,"third_edges_3 is hidden from the screen — tree left the board."],[77.69500000000001,"third_edges_4 is hidden from the screen — tree left the board."],[77.69500000000001,"third_edges_5 is hidden from the screen — tree left the board."],[77.69500000000001,"third_edges_6 is hidden from the screen — tree left the board."],[77.69500000000001,"third_edges_7 is hidden from the screen — tree left the board."],[77.69500000000001,"third_edges_8 is hidden from the screen — tree left the board."]]},{"start":78.295,"say":"We give one balanced binary answer a unit of information and call that unit a bit. A bit is not merely a zero stored in memory. It is the distinction between two alternatives that were both still possible.","live":[],"does":[[83.009,"bit_definition is shown on the screen, written out."]]},{"start":92.07300000000001,"say":"With q binary answers there are two to the q possible answer strings. Therefore identifying one of N equally likely outcomes requires log base two of N bits.","live":["bit_definition"],"does":[[92.66499999999999,"binary_note is shown on the screen, written out."],[94.325,"binary_note (the \"2^q\" part) is emphasized."],[102.998,"binary_note is hidden from the screen — left the board."],[102.998,"bit_definition is hidden from the screen — left the board."],[102.998,"binary_note (the \"2^q\" part) is no longer emphasized."]]},{"start":103.59800000000001,"say":"Next we will make the letters unequally likely. That one change will lead us first to entropy, the ideal average question count, and then to Huffman coding, a way to build an actual binary description close to that ideal.","live":[],"does":[[109.26400000000001,"destination is shown on the screen, written out."],[109.26400000000001,"destination (the \"entropy\" part) is emphasized."],[112.932,"destination (the \"Huffman coding\" part) is emphasized."],[112.932,"destination (the \"entropy\" part) is no longer emphasized."],[117.96162500000001,"destination (the \"Huffman coding\" part) is no longer emphasized."],[118.21162500000001,"destination is hidden from the screen — left the board."]]}]},{"title":"Likelihood Changes the Questions","start":119.25329166666668,"end":255.6412916666667,"objects":{"bar":"a Figure (x_range=(0.0, 100.0), y_range=(0.0, 1.5), aspect=(10.0, 2.0))","comparison":"a Math [text] that says \"$3.00 - 2.60 = 0.40 thin upright(\"question saved on average\")$\"","distribution":"a Table [text] that says \"Symbol Probability A 40% B 20% C 10% D 10% E 5% F 5% G 5% H 5%\" (rows=(('Symbol', 'Probability'), ('A', '40%'), ('B', '20%'), ('C', '…, header=True)","expected_work":"a Derivation [text] that says \"$L = 0.4(1) + 0.2(2) + 0.2(4) + 0.2(5) \\ L = 2.60 thin upright(\"questions\")$\"","heading_distribution":"a Heading that says \"The Same Symbols, Different Likelihoods\"","heading_plan":"a Heading that says \"Spend Short Answers on Likely Symbols\"","labels":"a Point [red] labelled \"A thin 40%\" drawn in bar (location=(20.0, 0.5), show_marker=False)","labels_2":"a Point [blue] labelled \"B thin 20%\" drawn in bar (location=(50.0, 0.5), show_marker=False)","labels_3":"a Point [green] labelled \"C thin 10%\" drawn in bar (location=(65.0, 0.5), show_marker=False)","labels_4":"a Point [green] labelled \"D thin 10%\" drawn in bar (location=(75.0, 0.5), show_marker=False)","labels_5":"a Point [yellow] labelled \"E thin 5%\" drawn in bar (location=(82.5, 0.5), show_marker=False)","labels_6":"a Point [yellow] labelled \"F thin 5%\" drawn in bar (location=(87.5, 0.5), show_marker=False)","labels_7":"a Point [yellow] labelled \"G thin 5%\" drawn in bar (location=(92.5, 0.5), show_marker=False)","labels_8":"a Point [yellow] labelled \"H thin 5%\" drawn in bar (location=(97.5, 0.5), show_marker=False)","plan":"a Table [text] that says \"Symbol $p$ $q$ A 40% 1 B 20% 2 C 10% 4 D 10% 4 E 5% 5 F 5% 5 G 5% 5 H 5% 5\" (rows=(('Symbol', '$p$', '$q$'), ('A', '40%', '1'), ('B', '20%', '2')…, header=True)","regions":"a Polygon [red] drawn in bar (vertices=((0.0, 0.0), (40.0, 0.0), (40.0, 1.0), (0.0, 1.0)), fill_opacity=0.45)","regions_2":"a Polygon [blue] drawn in bar (vertices=((40.0, 0.0), (60.0, 0.0), (60.0, 1.0), (40.0, 1.0)), fill_opacity=0.45)","regions_3":"a Polygon [green] drawn in bar (vertices=((60.0, 0.0), (70.0, 0.0), (70.0, 1.0), (60.0, 1.0)), fill_opacity=0.45)","regions_4":"a Polygon [green] drawn in bar (vertices=((70.0, 0.0), (80.0, 0.0), (80.0, 1.0), (70.0, 1.0)), fill_opacity=0.45)","regions_5":"a Polygon [yellow] drawn in bar (vertices=((80.0, 0.0), (85.0, 0.0), (85.0, 1.0), (80.0, 1.0)), fill_opacity=0.45)","regions_6":"a Polygon [yellow] drawn in bar (vertices=((85.0, 0.0), (90.0, 0.0), (90.0, 1.0), (85.0, 1.0)), fill_opacity=0.45)","regions_7":"a Polygon [yellow] drawn in bar (vertices=((90.0, 0.0), (95.0, 0.0), (95.0, 1.0), (90.0, 1.0)), fill_opacity=0.45)","regions_8":"a Polygon [yellow] drawn in bar (vertices=((95.0, 0.0), (100.0, 0.0), (100.0, 1.0), (95.0, 1.0)), fill_opacity=0.45)"},"beats":[{"start":119.25329166666668,"say":"Keep the same eight symbols, but change their likelihoods. A now occurs forty percent of the time. B occurs twenty percent. C and D each occur ten percent, and the final four symbols each occur five percent.","live":[],"does":[[119.25329166666668,"heading_distribution is shown on the screen, written out."],[119.25329166666668,"distribution is shown on the screen, written out."],[121.44729166666669,"bar is shown on the screen, written out."]]},{"start":134.74879166666668,"say":"A alone occupies forty percent of the probability bar. It is twice as likely as B, and four times as likely as either C or D.","live":["bar","heading_distribution"],"does":[[135.15529166666667,"distribution is shown on the screen, written out."],[136.68729166666668,"regions is shown on the screen, written out."],[136.68729166666668,"labels is shown on the screen, written out."],[140.18229166666669,"regions_2 is shown on the screen, written out."],[140.18229166666669,"labels_2 is shown on the screen, written out."],[140.18229166666669,"distribution is shown on the screen, written out."]]},{"start":144.4042916666667,"say":"C and D together occupy another twenty percent. They are less common individually, but they still matter much more often than any one of the four symbols at the narrow end.","live":["bar","heading_distribution","regions","labels","regions_2","labels_2"],"does":[[144.75229166666668,"regions_3 is shown on the screen, written out."],[144.75229166666668,"labels_3 is shown on the screen, written out."],[144.75229166666668,"distribution is shown on the screen, written out."],[145.24029166666668,"regions_4 is shown on the screen, written out."],[145.24029166666668,"labels_4 is shown on the screen, written out."],[145.24029166666668,"distribution is shown on the screen, written out."]]},{"start":155.20929166666667,"say":"The final four slices are equal and rare. Each accounts for only one twentieth of the messages.","live":["bar","heading_distribution","regions","labels","regions_2","labels_2","regions_3","labels_3","regions_4","labels_4"],"does":[[156.11529166666668,"regions_5 is shown on the screen, written out."],[156.11529166666668,"labels_5 is shown on the screen, written out."],[156.11529166666668,"regions_6 is shown on the screen, written out."],[156.11529166666668,"labels_6 is shown on the screen, written out."],[156.11529166666668,"regions_7 is shown on the screen, written out."],[156.11529166666668,"labels_7 is shown on the screen, written out."],[156.11529166666668,"regions_8 is shown on the screen, written out."],[156.11529166666668,"labels_8 is shown on the screen, written out."],[156.11529166666668,"distribution is shown on the screen, written out."],[156.11529166666668,"distribution is shown on the screen, written out."],[156.11529166666668,"distribution is shown on the screen, written out."],[156.11529166666668,"distribution is shown on the screen, written out."]]},{"start":162.33429166666667,"say":"If we keep the old balanced tree, every outcome still costs three questions. That treats a five-percent event exactly like the event that appears forty percent of the time.","live":["bar","heading_distribution","regions","labels","regions_2","labels_2","regions_3","labels_3","regions_4","labels_4","regions_5","labels_5","regions_6","labels_6","regions_7","labels_7","regions_8","labels_8"],"does":[[168.7432916666667,"distribution (the \"row=6\" part) is emphasized."],[171.48329166666667,"distribution (the \"row=2\" part) is emphasized."],[171.48329166666667,"distribution (the \"row=6\" part) is no longer emphasized."],[173.1432916666667,"bar is hidden from the screen — left the board."],[173.1432916666667,"regions is hidden from the screen — bar left the board."],[173.1432916666667,"labels is hidden from the screen — bar left the board."],[173.1432916666667,"regions_2 is hidden from the screen — bar left the board."],[173.1432916666667,"labels_2 is hidden from the screen — bar left the board."],[173.1432916666667,"regions_3 is hidden from the screen — bar left the board."],[173.1432916666667,"labels_3 is hidden from the screen — bar left the board."],[173.1432916666667,"regions_4 is hidden from the screen — bar left the board."],[173.1432916666667,"labels_4 is hidden from the screen — bar left the board."],[173.1432916666667,"regions_5 is hidden from the screen — bar left the board."],[173.1432916666667,"labels_5 is hidden from the screen — bar left the board."],[173.1432916666667,"regions_6 is hidden from the screen — bar left the board."],[173.1432916666667,"labels_6 is hidden from the screen — bar left the board."],[173.1432916666667,"regions_7 is hidden from the screen — bar left the board."],[173.1432916666667,"labels_7 is hidden from the screen — bar left the board."],[173.1432916666667,"regions_8 is hidden from the screen — bar left the board."],[173.1432916666667,"labels_8 is hidden from the screen — bar left the board."],[173.1432916666667,"distribution is hidden from the screen — left the board."],[173.1432916666667,"heading_distribution is hidden from the screen — left the board."],[173.1432916666667,"distribution (the \"row=2\" part) is no longer emphasized."]]},{"start":173.7432916666667,"say":"Instead, ask the most useful question first: is the symbol A? When the answer is yes, which happens forty percent of the time, the search ends after one bit.","live":[],"does":[[173.7432916666667,"heading_plan is shown on the screen, written out."],[173.7432916666667,"plan is shown on the screen, written out."],[178.17829166666667,"plan is shown on the screen, written out."],[183.65829166666668,"plan (the \"1\" part) is indicated — a transient flash."]]},{"start":185.1982916666667,"say":"If it was not A, the next question can separate B. B then takes two answers. The commonest sixty percent of all symbols have received the two shortest paths.","live":["heading_plan"],"does":[[188.1592916666667,"plan is shown on the screen, written out."],[190.2022916666667,"plan (the \"2\" part) is emphasized."],[195.9147916666667,"plan (the \"2\" part) is no longer emphasized."]]},{"start":196.5147916666667,"say":"The price is that C and D now require four questions. That sounds worse than the old three, but each of those symbols appears only one time in ten.","live":null,"does":[[197.7102916666667,"plan is shown on the screen, written out."],[198.1512916666667,"plan is shown on the screen, written out."],[199.0802916666667,"plan (the \"row=4\" part) is emphasized."],[199.0802916666667,"plan (the \"row=5\" part) is emphasized."],[203.5732916666667,"plan (the \"row=4\" part) is no longer emphasized."],[205.9652916666667,"plan (the \"row=5\" part) is no longer emphasized."]]},{"start":206.5652916666667,"say":"The four rarest outcomes take five questions each. Long answers are not free, but we pay for them on only twenty percent of the symbols.","live":null,"does":[[207.0872916666667,"plan is shown on the screen, written out."],[207.0872916666667,"plan is shown on the screen, written out."],[207.0872916666667,"plan is shown on the screen, written out."],[207.0872916666667,"plan is shown on the screen, written out."],[208.4922916666667,"plan (the \"column=3\" part) is emphasized."],[215.0987916666667,"plan (the \"column=3\" part) is no longer emphasized."]]},{"start":215.6987916666667,"say":"Now average the question counts using their probabilities. A contributes point four times one. B contributes point two times two. C and D together contribute point two times four, and the rare group contributes point two times five.","live":null,"does":[[216.2562916666667,"expected_work is shown on the screen, written out."],[219.40229166666668,"expected_work (the \"0.4(1)\" part) is emphasized."],[221.88729166666667,"expected_work (the \"0.2(2)\" part) is emphasized."],[221.88729166666667,"expected_work (the \"0.4(1)\" part) is no longer emphasized."],[224.44129166666667,"expected_work (the \"0.2(2)\" part) is no longer emphasized."],[224.44129166666667,"expected_work (the \"0.2(4)\" part) is emphasized."],[227.93529166666667,"expected_work (the \"0.2(4)\" part) is no longer emphasized."],[227.93529166666667,"expected_work (the \"0.2(5)\" part) is emphasized."],[230.49029166666668,"expected_work (the \"0.2(5)\" part) is no longer emphasized."]]},{"start":231.0902916666667,"say":"The result is two point six questions per symbol. Some individual outcomes became more expensive, but the outcomes we see most often became much cheaper.","live":null,"does":[[231.6362916666667,"expected_work is shown on the screen, written out."],[232.20529166666668,"expected_work (the \"2.60\" part) is indicated — a transient flash."]]},{"start":241.10579166666668,"say":"The equal-depth plan cost exactly three questions. Exploiting likelihood saves point four of a question per symbol on average. Over a million symbols, that becomes roughly four hundred thousand fewer binary answers.","live":null,"does":[[246.08629166666668,"comparison is shown on the screen, written out."],[248.4782916666667,"A box is drawn around comparison."],[254.59962500000003,"comparison is hidden from the screen — left the board."],[254.59962500000003,"expected_work is hidden from the screen — left the board."],[254.59962500000003,"heading_plan is hidden from the screen — left the board."],[254.59962500000003,"plan is hidden from the screen — left the board."]]}]},{"title":"From Question Counts to Entropy","start":255.6412916666667,"end":417.7835625,"objects":{"block_bits":"a Math [text] that says \"$log_2 2^(n H) = n H thin upright(\"bits\")$\"","block_count":"a Math [text] that says \"$2^(n H) thin upright(\"typical length-n messages\")$\"","c2":"a Math [text] that says \"$I(p) = -log_2 p$\"","contributions":"a Table [text] that says \"Group Probability each Ideal bits Total contribution A 0.40 1.322 0.529 B 0.20 2.322 0.464 C, D 0.10 3.322 0.664 E, F, G, H 0.05 4.322 0.864\" (rows=(('Group', 'Probability each', 'Ideal bits', 'Total contributio…, header=True)","count_derivation":"a Derivation [text] that says \"$2^q approx frac(1, p) \\ q approx log_2 frac(1, p) \\ I(p) = -log_2 p$\"","entropy_formula":"a Math [text] that says \"$H = sum_i p_i log_2 frac(1, p_i)$\"","entropy_value":"a Math [text] that says \"$H approx 2.522 thin upright(\"bits per symbol\")$\"","fractional_note":"a Text [text] that says \"An ideal length may be fractional. It is a long-run average target, not a literal fraction of one yes-or-no answer.\"","heading_blocks":"a Heading that says \"Count Long Messages Instead of Single Symbols\"","heading_entropy":"a Heading that says \"Average the Ideal Question Count\"","heading_ideal":"a Heading that says \"How Many Halves Reach Probability $p$?\"","mass":"a Figure (x_range=(0.0, 100.0), y_range=(0.0, 1.5), aspect=(10.0, 2.0))","pieces":"a Polygon [red] drawn in mass (vertices=((0.0, 0.0), (40.0, 0.0), (40.0, 1.0), (0.0, 1.0)), fill_opacity=0.45)","pieces_2":"a Polygon [blue] drawn in mass (vertices=((40.0, 0.0), (60.0, 0.0), (60.0, 1.0), (40.0, 1.0)), fill_opacity=0.45)","pieces_3":"a Polygon [green] drawn in mass (vertices=((60.0, 0.0), (70.0, 0.0), (70.0, 1.0), (60.0, 1.0)), fill_opacity=0.45)","pieces_4":"a Polygon [green] drawn in mass (vertices=((70.0, 0.0), (80.0, 0.0), (80.0, 1.0), (70.0, 1.0)), fill_opacity=0.45)","pieces_5":"a Polygon [yellow] drawn in mass (vertices=((80.0, 0.0), (85.0, 0.0), (85.0, 1.0), (80.0, 1.0)), fill_opacity=0.45)","pieces_6":"a Polygon [yellow] drawn in mass (vertices=((85.0, 0.0), (90.0, 0.0), (90.0, 1.0), (85.0, 1.0)), fill_opacity=0.45)","pieces_7":"a Polygon [yellow] drawn in mass (vertices=((90.0, 0.0), (95.0, 0.0), (95.0, 1.0), (90.0, 1.0)), fill_opacity=0.45)","pieces_8":"a Polygon [yellow] drawn in mass (vertices=((95.0, 0.0), (100.0, 0.0), (100.0, 1.0), (95.0, 1.0)), fill_opacity=0.45)"},"beats":[{"start":255.6412916666667,"say":"For eight equal outcomes, three halvings isolate one outcome because one eighth equals one over two cubed. Now replace one eighth by a general probability p.","live":[],"does":[[255.6412916666667,"heading_ideal is shown on the screen, written out."],[255.6412916666667,"mass is shown on the screen, written out."],[264.2212916666667,"pieces is shown on the screen, written out."],[264.2212916666667,"pieces_2 is shown on the screen, written out."],[264.2212916666667,"pieces_3 is shown on the screen, written out."],[264.2212916666667,"pieces_4 is shown on the screen, written out."],[264.2212916666667,"pieces_5 is shown on the screen, written out."],[264.2212916666667,"pieces_6 is shown on the screen, written out."],[264.2212916666667,"pieces_7 is shown on the screen, written out."],[264.2212916666667,"pieces_8 is shown on the screen, written out."]]},{"start":266.1212916666667,"say":"Imagine a very large collection of equally detailed possibilities. An event with probability p occupies the fraction p of them. To isolate that fraction by balanced binary questions, we keep halving.","live":["mass","heading_ideal","pieces","pieces_2","pieces_3","pieces_4","pieces_5","pieces_6","pieces_7","pieces_8"],"does":[[273.2732916666667,"pieces is indicated — a transient flash."]]},{"start":280.1427916666667,"say":"After q balanced questions, one answer path identifies about one part in two to the q. To match an event occupying fraction p, two to the q must be about one over p.","live":null,"does":[[284.88029166666666,"mass moves to a new place on the board."],[284.88029166666666,"count_derivation is shown on the screen, written out."],[284.88029166666666,"count_derivation (the \"2^q\" part) is indicated — a transient flash."],[291.3232916666667,"count_derivation (the \"frac(1, p)\" part) is indicated — a transient flash."]]},{"start":293.3282916666667,"say":"Solve for q. The ideal question count is log base two of one over p. Equivalently, it is minus log base two of p.","live":null,"does":[[293.6652916666667,"count_derivation is shown on the screen, written out."],[298.87829166666666,"count_derivation is shown on the screen, written out."]]},{"start":303.0247916666667,"say":"This quantity measures surprise. A likely event has a large p and a small information value. A rare event has a small p and needs a longer answer path to distinguish it.","live":null,"does":[[305.85129166666667,"pieces is indicated — a transient flash."],[310.5542916666667,"pieces_8 is indicated — a transient flash."]]},{"start":315.9372916666667,"say":"For A, p is point four, so the ideal length is about one point three two two bits. For a five-percent symbol, it is about four point three two two bits. These are not yet legal single-symbol code lengths.","live":null,"does":[[318.7472916666667,"count_derivation is indicated — a transient flash."],[330.7402916666667,"count_derivation is hidden from the screen — left the board."],[330.7402916666667,"heading_ideal is hidden from the screen — left the board."],[330.7402916666667,"mass is hidden from the screen — left the board."],[330.7402916666667,"pieces is hidden from the screen — mass left the board."],[330.7402916666667,"pieces_2 is hidden from the screen — mass left the board."],[330.7402916666667,"pieces_3 is hidden from the screen — mass left the board."],[330.7402916666667,"pieces_4 is hidden from the screen — mass left the board."],[330.7402916666667,"pieces_5 is hidden from the screen — mass left the board."],[330.7402916666667,"pieces_6 is hidden from the screen — mass left the board."],[330.7402916666667,"pieces_7 is hidden from the screen — mass left the board."],[330.7402916666667,"pieces_8 is hidden from the screen — mass left the board."]]},{"start":331.94029166666667,"say":"A single binary question cannot be split into a fraction. But across a long stream, common and rare symbols can share a description strategy, and a fractional average becomes meaningful.","live":[],"does":[[331.94029166666667,"heading_entropy is shown on the screen, written out."],[341.8082916666667,"fractional_note is shown on the screen, written out."]]},{"start":344.8927916666667,"say":"To find the ideal average, weight each symbol's question count by the probability of seeing that symbol, then add. This weighted average is the quantity we call entropy.","live":["fractional_note","heading_entropy"],"does":[[346.5302916666667,"entropy_formula is shown on the screen, written out."],[347.4592916666667,"entropy_formula (the \"log_2 frac(1, p_i)\" part) is emphasized."],[348.4572916666667,"entropy_formula (the \"log_2 frac(1, p_i)\" part) is no longer emphasized."],[348.4572916666667,"entropy_formula (the \"p_i\" part) is emphasized."],[354.93579166666666,"entropy_formula (the \"p_i\" part) is no longer emphasized."]]},{"start":355.5357916666667,"say":"Now calculate it for our distribution. A contributes about point five two nine bits per source symbol. That is its ideal length multiplied by the forty-percent frequency with which we pay it.","live":["entropy_formula","fractional_note","heading_entropy"],"does":[[356.1512916666667,"contributions is shown on the screen, written out."],[358.4732916666667,"contributions is shown on the screen, written out."],[359.5642916666667,"contributions (the \"0.529\" part) is emphasized."],[367.0992916666667,"contributions (the \"0.529\" part) is no longer emphasized."]]},{"start":367.6992916666667,"say":"B contributes about point four six four. C and D together contribute about point six six four.","live":null,"does":[[368.0472916666667,"contributions is shown on the screen, written out."],[369.3602916666667,"contributions (the \"0.464\" part) is indicated — a transient flash."],[371.1822916666667,"contributions is shown on the screen, written out."],[373.4702916666667,"contributions (the \"0.664\" part) is indicated — a transient flash."]]},{"start":375.5327916666667,"say":"The four five-percent symbols each carry more surprise, and together they contribute about point eight six four bits. Adding every group gives an entropy of about two point five two two bits per symbol.","live":null,"does":[[376.06729166666673,"contributions is shown on the screen, written out."],[385.7492916666667,"entropy_value is shown on the screen, written out."],[387.5262916666667,"A box is drawn around entropy_value."],[388.5012916666667,"contributions is hidden from the screen — left the board."],[388.5012916666667,"entropy_formula is hidden from the screen — left the board."],[388.5012916666667,"entropy_value is hidden from the screen — left the board."],[388.5012916666667,"fractional_note is hidden from the screen — left the board."],[388.5012916666667,"heading_entropy is hidden from the screen — left the board."]]},{"start":389.7012916666667,"say":"Here is another way to read the same number. In a long block of n symbols, the probability pattern concentrates most messages among roughly two to the n H typical sequences.","live":[],"does":[[389.7012916666667,"heading_blocks is shown on the screen, written out."],[398.4082916666667,"block_count is shown on the screen, written out."]]},{"start":401.7027916666667,"say":"Distinguishing that many likely messages needs about the log of their count, which is n H binary answers. Entropy is therefore not an arbitrary formula. It is the question count forced by the number of likely messages.","live":["block_count","heading_blocks"],"does":[[405.2552916666667,"block_bits is shown on the screen, written out."],[406.7992916666667,"A box is drawn around block_bits."],[416.7418958333334,"block_bits is hidden from the screen — left the board."],[416.7418958333334,"block_count is hidden from the screen — left the board."],[416.7418958333334,"heading_blocks is hidden from the screen — left the board."]]}]},{"title":"Build the Huffman Tree","start":417.7835625,"end":569.1103333333333,"objects":{"average_work":"a Derivation [text] that says \"$L = 0.40(1) + 0.20(2) + 0.20(4) + 0.20(5) \\ L = 2.60 thin upright(\"bits per symbol\")$\"","codes":"a Table [text] that says \"Symbol Probability Code Length A 0.40 0 1 B 0.20 10 2 C 0.10 1100 4 D 0.10 1101 4 E 0.05 11100 5 F 0.05 11101 5 G 0.05 11110 5 H 0.05 11111 5\" (rows=(('Symbol', 'Probability', 'Code', 'Length'), ('A', '0.40', '0'…, header=True)","edge_40":"a Line [yellow] drawn in tree (start=(3.35, 2.05), end=(4.75, 3.0))","edge_40_2":"a Line [yellow] drawn in tree (start=(3.35, 2.05), end=(4.75, 1.25))","edge_60":"a Line [yellow] drawn in tree (start=(2.0, 2.55), end=(3.35, 4.05))","edge_60_2":"a Line [yellow] drawn in tree (start=(2.0, 2.55), end=(3.35, 2.05))","edge_cd":"a Line [yellow] drawn in tree (start=(4.75, 3.0), end=(6.05, 3.45))","edge_cd_2":"a Line [yellow] drawn in tree (start=(4.75, 3.0), end=(6.05, 2.65))","edge_ef":"a Line [yellow] drawn in tree (start=(6.05, 1.65), end=(7.45, 1.95))","edge_ef_2":"a Line [yellow] drawn in tree (start=(6.05, 1.65), end=(7.45, 1.35))","edge_efgh":"a Line [yellow] drawn in tree (start=(4.75, 1.25), end=(6.05, 1.65))","edge_efgh_2":"a Line [yellow] drawn in tree (start=(4.75, 1.25), end=(6.05, 0.75))","edge_gh":"a Line [yellow] drawn in tree (start=(6.05, 0.75), end=(7.45, 0.85))","edge_gh_2":"a Line [yellow] drawn in tree (start=(6.05, 0.75), end=(7.45, 0.25))","edge_root":"a Line [red] drawn in tree (start=(0.55, 3.0), end=(2.0, 5.35))","edge_root_2":"a Line [red] drawn in tree (start=(0.55, 3.0), end=(2.0, 2.55))","heading_build":"a Heading that says \"Merge the Two Rarest, Again and Again\"","heading_code":"a Heading that says \"Read Zero and One from Root to Leaf\"","leaf_a":"a Point [blue] labelled \"A thin 40%\" drawn in tree (location=(2.0, 5.35))","leaf_b":"a Point [blue] labelled \"B thin 20%\" drawn in tree (location=(3.35, 4.05))","leaf_c":"a Point [blue] labelled \"C thin 10%\" drawn in tree (location=(6.05, 3.45))","leaf_d":"a Point [blue] labelled \"D thin 10%\" drawn in tree (location=(6.05, 2.65))","leaf_e":"a Point [blue] labelled \"E thin 5%\" drawn in tree (location=(7.45, 1.95))","leaf_f":"a Point [blue] labelled \"F thin 5%\" drawn in tree (location=(7.45, 1.35))","leaf_g":"a Point [blue] labelled \"G thin 5%\" drawn in tree (location=(7.45, 0.85))","leaf_h":"a Point [blue] labelled \"H thin 5%\" drawn in tree (location=(7.45, 0.25))","merges":"a Table [text] that says \"Step Two smallest weights Merged weight 1 E 5 + F 5 10 2 G 5 + H 5 10 3 C 10 + D 10 20 4 10 + 10 20 5 20 + 20 40 6 B 20 + 40 60 7 A 40 + 60 100\" (rows=(('Step', 'Two smallest weights', 'Merged weight'), ('1', 'E 5 …, header=True)","node_40":"a Point [yellow] labelled \"40\" drawn in tree (location=(3.35, 2.05))","node_60":"a Point [yellow] labelled \"60\" drawn in tree (location=(2.0, 2.55))","node_cd":"a Point [yellow] labelled \"20\" drawn in tree (location=(4.75, 3.0))","node_ef":"a Point [yellow] labelled \"10\" drawn in tree (location=(6.05, 1.65))","node_efgh":"a Point [yellow] labelled \"20\" drawn in tree (location=(4.75, 1.25))","node_gh":"a Point [yellow] labelled \"10\" drawn in tree (location=(6.05, 0.75))","node_root":"a Point [red] labelled \"100\" drawn in tree (location=(0.55, 3.0))","tree":"a Figure (x_range=(0.0, 8.2), y_range=(0.0, 6.0), aspect=(4.0, 3.0))"},"beats":[{"start":417.7835625,"say":"Huffman coding turns the probability list into a binary tree. Begin with one leaf for every symbol. The only repeated instruction is to merge the two smallest weights.","live":[],"does":[[417.7835625,"heading_build is shown on the screen, written out."],[417.7835625,"merges is shown on the screen, written out."],[421.09256250000004,"tree is shown on the screen, written out."],[423.1125625,"leaf_a is shown on the screen, written out."],[423.1125625,"leaf_b is shown on the screen, written out."],[423.1125625,"leaf_c is shown on the screen, written out."],[423.1125625,"leaf_d is shown on the screen, written out."],[423.1125625,"leaf_e is shown on the screen, written out."],[423.1125625,"leaf_f is shown on the screen, written out."],[423.1125625,"leaf_g is shown on the screen, written out."],[423.1125625,"leaf_h is shown on the screen, written out."]]},{"start":429.55206250000003,"say":"The first two rarest leaves are E and F, each with weight five. Join them under a new node of weight ten.","live":["tree","heading_build","leaf_a","leaf_b","leaf_c","leaf_d","leaf_e","leaf_f","leaf_g","leaf_h"],"does":[[430.0865625,"merges is shown on the screen, written out."],[434.6485625,"edge_ef is shown on the screen, written out."],[434.6485625,"edge_ef_2 is shown on the screen, written out."],[437.17956250000003,"node_ef is shown on the screen, written out."]]},{"start":438.58106250000003,"say":"G and H are the next equal pair. Merge their five-percent weights into a second node of weight ten.","live":["tree","heading_build","leaf_a","leaf_b","leaf_c","leaf_d","leaf_e","leaf_f","leaf_g","leaf_h","node_ef","edge_ef","edge_ef_2"],"does":[[438.92956250000003,"merges is shown on the screen, written out."],[442.0985625,"edge_gh is shown on the screen, written out."],[442.0985625,"edge_gh_2 is shown on the screen, written out."],[445.15256250000004,"node_gh is shown on the screen, written out."]]},{"start":446.4835625,"say":"C and D each have weight ten. Merge those two leaves into a node of weight twenty.","live":["tree","heading_build","leaf_a","leaf_b","leaf_c","leaf_d","leaf_e","leaf_f","leaf_g","leaf_h","node_ef","edge_ef","edge_ef_2","node_gh","edge_gh","edge_gh_2"],"does":[[446.7855625,"merges is shown on the screen, written out."],[449.8155625,"edge_cd is shown on the screen, written out."],[449.8155625,"edge_cd_2 is shown on the screen, written out."],[452.2185625,"node_cd is shown on the screen, written out."]]},{"start":453.7360625,"say":"The two ten-weight branches made from E through H are now the smallest pair. Their merge produces another weight twenty.","live":["tree","heading_build","leaf_a","leaf_b","leaf_c","leaf_d","leaf_e","leaf_f","leaf_g","leaf_h","node_ef","edge_ef","edge_ef_2","node_gh","edge_gh","edge_gh_2","node_cd","edge_cd","edge_cd_2"],"does":[[454.24656250000004,"merges is shown on the screen, written out."],[459.15756250000004,"edge_efgh is shown on the screen, written out."],[459.15756250000004,"edge_efgh_2 is shown on the screen, written out."],[460.4925625,"node_efgh is shown on the screen, written out."]]},{"start":462.0100625,"say":"Three weights of twenty are available. Ties may be resolved consistently in more than one way. Merge these two compound branches to make forty.","live":["tree","heading_build","leaf_a","leaf_b","leaf_c","leaf_d","leaf_e","leaf_f","leaf_g","leaf_h","node_ef","edge_ef","edge_ef_2","node_gh","edge_gh","edge_gh_2","node_cd","edge_cd","edge_cd_2","node_efgh","edge_efgh","edge_efgh_2"],"does":[[468.5815625,"merges is shown on the screen, written out."],[470.6475625,"node_40 is shown on the screen, written out."],[470.6475625,"edge_40 is shown on the screen, written out."],[470.6475625,"edge_40_2 is shown on the screen, written out."]]},{"start":472.10706250000004,"say":"B has weight twenty and the compound branch has weight forty. They are the two smallest remaining items, so their parent has weight sixty.","live":["tree","heading_build","leaf_a","leaf_b","leaf_c","leaf_d","leaf_e","leaf_f","leaf_g","leaf_h","node_ef","edge_ef","edge_ef_2","node_gh","edge_gh","edge_gh_2","node_cd","edge_cd","edge_cd_2","node_efgh","edge_efgh","edge_efgh_2","node_40","edge_40","edge_40_2"],"does":[[472.45556250000004,"merges is shown on the screen, written out."],[479.2475625,"edge_60 is shown on the screen, written out."],[479.2475625,"edge_60_2 is shown on the screen, written out."],[480.0595625,"node_60 is shown on the screen, written out."]]},{"start":481.54206250000004,"say":"Finally merge A, with weight forty, and the remaining weight sixty. Their root has weight one hundred, and the Huffman tree is complete.","live":["tree","heading_build","leaf_a","leaf_b","leaf_c","leaf_d","leaf_e","leaf_f","leaf_g","leaf_h","node_ef","edge_ef","edge_ef_2","node_gh","edge_gh","edge_gh_2","node_cd","edge_cd","edge_cd_2","node_efgh","edge_efgh","edge_efgh_2","node_40","edge_40","edge_40_2","node_60","edge_60","edge_60_2"],"does":[[481.8905625,"merges is shown on the screen, written out."],[482.3895625,"edge_root is shown on the screen, written out."],[482.3895625,"edge_root_2 is shown on the screen, written out."],[487.0915625,"node_root is shown on the screen, written out."],[490.59756250000004,"tree moves to a new place on the board."],[490.59756250000004,"heading_build is hidden from the screen — left the board."],[490.59756250000004,"merges is hidden from the screen — left the board."],[490.59756250000004,"codes is shown on the screen, written out."]]},{"start":491.7975625,"say":"Assign zero to the upper branch and one to the lower branch at each split. A codeword is the sequence encountered while walking from the root to a leaf.","live":["tree","leaf_a","leaf_b","leaf_c","leaf_d","leaf_e","leaf_f","leaf_g","leaf_h","node_ef","edge_ef","edge_ef_2","node_gh","edge_gh","edge_gh_2","node_cd","edge_cd","edge_cd_2","node_efgh","edge_efgh","edge_efgh_2","node_40","edge_40","edge_40_2","node_60","edge_60","edge_60_2","node_root","edge_root","edge_root_2"],"does":[[491.7975625,"heading_code is shown on the screen, written out."],[499.5655625,"node_root is indicated — a transient flash."]]},{"start":501.60456250000004,"say":"A is reached immediately by code zero, so the most likely symbol costs only one bit. B follows one then zero, so it costs two bits.","live":["tree","leaf_a","leaf_b","leaf_c","leaf_d","leaf_e","leaf_f","leaf_g","leaf_h","node_ef","edge_ef","edge_ef_2","node_gh","edge_gh","edge_gh_2","node_cd","edge_cd","edge_cd_2","node_efgh","edge_efgh","edge_efgh_2","node_40","edge_40","edge_40_2","node_60","edge_60","edge_60_2","node_root","edge_root","edge_root_2","heading_code"],"does":[[501.9525625,"codes is shown on the screen, written out."],[503.8565625,"codes (the \"0\" part) is emphasized."],[507.60756250000003,"codes is shown on the screen, written out."],[507.60756250000003,"codes (the \"0\" part) is no longer emphasized."],[508.3385625,"codes (the \"10\" part) is emphasized."],[511.1595625,"codes (the \"10\" part) is no longer emphasized."]]},{"start":511.7595625,"say":"C and D sit deeper, with codes one one zero zero and one one zero one. Each requires four bits.","live":null,"does":[[512.1075625,"codes is shown on the screen, written out."],[512.5495625,"codes is shown on the screen, written out."],[518.9225625,"codes (the \"column=4\" part) is indicated — a transient flash."]]},{"start":520.6020625,"say":"The four rarest symbols lie one level deeper still. Their codes share the prefix one one one, then use two more bits to select the leaf.","live":null,"does":[[521.1365625000001,"codes is shown on the screen, written out."],[521.1365625000001,"codes is shown on the screen, written out."],[521.1365625000001,"codes is shown on the screen, written out."],[521.1365625000001,"codes is shown on the screen, written out."],[525.3975625,"codes (the \"column=3\" part) is emphasized."],[529.3095625000001,"codes (the \"column=3\" part) is no longer emphasized."]]},{"start":529.9095625,"say":"No codeword is the beginning of another codeword. That prefix property lets a decoder read the stream from left to right and know exactly when each symbol ends.","live":null,"does":[[530.4675625,"codes (the \"0\" part) is indicated — a transient flash."],[531.5935625000001,"codes (the \"10\" part) is indicated — a transient flash."]]},{"start":539.9830625000001,"say":"Compute the average length. A pays one bit forty percent of the time. B pays two bits twenty percent of the time. C and D pay four, and the final four symbols pay five.","live":null,"does":[[540.9115625000001,"average_work is shown on the screen, written out."],[542.3045625,"average_work (the \"0.40(1)\" part) is emphasized."],[545.3465625,"average_work (the \"0.20(2)\" part) is emphasized."],[545.3465625,"average_work (the \"0.40(1)\" part) is no longer emphasized."],[548.6555625000001,"average_work (the \"0.20(2)\" part) is no longer emphasized."],[548.6555625000001,"average_work (the \"0.20(4)\" part) is emphasized."],[550.5245625,"average_work (the \"0.20(4)\" part) is no longer emphasized."],[550.5245625,"average_work (the \"0.20(5)\" part) is emphasized."],[552.6265625000001,"average_work (the \"0.20(5)\" part) is no longer emphasized."]]},{"start":553.2265625,"say":"The Huffman code therefore uses two point six bits per symbol on average. The merging rule has recovered the likelihood-aware question plan, and it has guaranteed that no other binary prefix code has a smaller average length for these individual symbols.","live":null,"does":[[555.3165625,"average_work is shown on the screen, written out."],[557.1625625,"A box is drawn around average_work."],[568.0686666666667,"average_work is hidden from the screen — left the board."],[568.0686666666667,"codes is hidden from the screen — left the board."],[568.0686666666667,"heading_code is hidden from the screen — left the board."],[568.0686666666667,"tree is hidden from the screen — left the board."],[568.0686666666667,"leaf_a is hidden from the screen — tree left the board."],[568.0686666666667,"leaf_b is hidden from the screen — tree left the board."],[568.0686666666667,"leaf_c is hidden from the screen — tree left the board."],[568.0686666666667,"leaf_d is hidden from the screen — tree left the board."],[568.0686666666667,"leaf_e is hidden from the screen — tree left the board."],[568.0686666666667,"leaf_f is hidden from the screen — tree left the board."],[568.0686666666667,"leaf_g is hidden from the screen — tree left the board."],[568.0686666666667,"leaf_h is hidden from the screen — tree left the board."],[568.0686666666667,"node_ef is hidden from the screen — tree left the board."],[568.0686666666667,"edge_ef is hidden from the screen — tree left the board."],[568.0686666666667,"edge_ef_2 is hidden from the screen — tree left the board."],[568.0686666666667,"node_gh is hidden from the screen — tree left the board."],[568.0686666666667,"edge_gh is hidden from the screen — tree left the board."],[568.0686666666667,"edge_gh_2 is hidden from the screen — tree left the board."],[568.0686666666667,"node_cd is hidden from the screen — tree left the board."],[568.0686666666667,"edge_cd is hidden from the screen — tree left the board."],[568.0686666666667,"edge_cd_2 is hidden from the screen — tree left the board."],[568.0686666666667,"node_efgh is hidden from the screen — tree left the board."],[568.0686666666667,"edge_efgh is hidden from the screen — tree left the board."],[568.0686666666667,"edge_efgh_2 is hidden from the screen — tree left the board."],[568.0686666666667,"node_40 is hidden from the screen — tree left the board."],[568.0686666666667,"edge_40 is hidden from the screen — tree left the board."],[568.0686666666667,"edge_40_2 is hidden from the screen — tree left the board."],[568.0686666666667,"node_60 is hidden from the screen — tree left the board."],[568.0686666666667,"edge_60 is hidden from the screen — tree left the board."],[568.0686666666667,"edge_60_2 is hidden from the screen — tree left the board."],[568.0686666666667,"node_root is hidden from the screen — tree left the board."],[568.0686666666667,"edge_root is hidden from the screen — tree left the board."],[568.0686666666667,"edge_root_2 is hidden from the screen — tree left the board."]]}]},{"title":"The Bound and the Files We Compress","start":569.1103333333333,"end":864.6787291666667,"objects":{"block_note":"a Text [text] that says \"Coding longer blocks makes the one-bit rounding allowance apply to an entire block, so its cost per source symbol can become arbitrarily small.\"","bound_result":"a Math [text] that says \"$L >= H$\"","bound_work":"a Derivation [text] that says \"$sum_i 2^(-l_i) <= 1 \\ L-H &= sum_i p_i log_2 frac(p_i, 2^(-l_i)) \\ &>= -log_2 (sum_i 2^(-l_i)) \\ &>= 0$\"","capacity_note":"a Text [text] that says \"A codeword of length $l_i$ occupies the fraction $2^(-l_i)$ of a complete binary tree. Prefix-free leaves cannot overlap.\"","english":"a Table [text] that says \"Property Typical English text Distribution Uneven Context Strong dependencies Redundancy Letters and phrases predictable Entropy Below 8 bits per byte Compression Usually substantial\" (rows=(('Property', 'Typical English text'), ('Distribution', 'Uneven…, header=True)","english_label":"a Tex [text] that says \"Structured text\"","entropy_point":"a Point [red] labelled \"H = 2.522\" drawn in scale (location=(2.522, 0.0))","entropy_value":"a Math [text] that says \"$H approx 2.522$\"","gap_value":"a Math [text] that says \"$L - H approx 0.078 thin upright(\"bit per symbol\")$\"","heading":"a Heading that says \"Why Huffman Gets So Close\"","heading_2":"a Heading that says \"Why Random Data Has No General Shortcut\"","heading_bound":"a Heading that says \"Why a Lossless Prefix Code Cannot Beat $H$\"","heading_compare":"a Heading that says \"Ideal Limit and Achievable Code\"","heading_files":"a Heading that says \"Why Some Files Shrink and Others Do Not\"","heading_recap":"a Heading that says \"The Compression Story\"","huffman_bound":"a Math [text] that says \"$H <= L_(upright(\"Huffman\")) < H + 1$\"","huffman_point":"a Point [green] labelled \"L = 2.600\" drawn in scale (location=(2.6, 0.0))","huffman_value":"a Math [text] that says \"$L_(upright(\"Huffman\")) = 2.600$\"","pigeonhole":"a Math [text] that says \"$2^(8n) thin upright(\"input files\") > sum_(j=0)^(8n-1) 2^j thin upright(\"shorter strings\")$\"","random":"a Table [text] that says \"Property Independent random bytes Distribution Uniform over 256 values Context No predictive dependence Redundancy None on average Entropy 8 bits per byte Compression None on average\" (rows=(('Property', 'Independent random bytes'), ('Distribution', 'Un…, header=True)","random_count":"a Math [text] that says \"$256 = 2^8$\"","random_entropy":"a Math [text] that says \"$H = log_2 256 = 8 thin upright(\"bits per byte\")$\"","random_label":"a Tex [text] that says \"Uniform random bytes\"","recap_four":"a Text [text] that says \"4. Predictability creates the room that compression uses.\"","recap_one":"a Text [text] that says \"1. A bit answers one binary question.\"","recap_three":"a Text [text] that says \"3. Huffman coding gives likely symbols shorter paths.\"","recap_two":"a Text [text] that says \"2. Entropy is the ideal average number of such questions.\"","scale":"a NumberLine labelled \"upright(\"bits per symbol\")\" (x_range=(2.4, 3.0), include_numbers=True, ticks_every=0.1)"},"beats":[{"start":569.1103333333333,"say":"Put the two numbers on one scale. Entropy for our source is about two point five two two bits per symbol. The Huffman code uses two point six.","live":[],"does":[[569.1103333333333,"heading_compare is shown on the screen, written out."],[570.4103333333333,"scale is shown on the screen, written out."],[571.7113333333333,"entropy_point is shown on the screen, written out."],[573.2323333333333,"scale moves to a new place on the board."],[573.2323333333333,"entropy_value is shown on the screen, written out."],[575.9483333333333,"huffman_point is shown on the screen, written out."],[577.1093333333333,"huffman_value is shown on the screen, written out."]]},{"start":578.9983333333333,"say":"The gap is only point zero seven eight of a bit per symbol. Huffman coding is not merely better than the fixed three-bit code. It is very close to the information-theoretic target.","live":["entropy_value","huffman_value","scale","heading_compare","entropy_point","huffman_point"],"does":[[579.4163333333333,"gap_value is shown on the screen, written out."],[587.6713333333333,"The segment (2.522, 0.0) to (2.6, 0.0) in scale is lit up."],[590.1323333333333,"scale: retire a lit segment (unemphasize_line)."]]},{"start":590.7323333333333,"say":"Two questions remain. Why can no lossless prefix code move the green point to the left of entropy? And if the bound is unbeatable, why is Huffman allowed to sit slightly above it?","live":["entropy_value","huffman_value","gap_value","scale","heading_compare","entropy_point","huffman_point"],"does":[[603.1088333333333,"entropy_value is hidden from the screen — left the board."],[603.1088333333333,"gap_value is hidden from the screen — left the board."],[603.1088333333333,"heading_compare is hidden from the screen — left the board."],[603.1088333333333,"huffman_value is hidden from the screen — left the board."],[603.1088333333333,"scale is hidden from the screen — left the board."],[603.1088333333333,"entropy_point is hidden from the screen — scale left the board."],[603.1088333333333,"huffman_point is hidden from the screen — scale left the board."]]},{"start":604.3088333333333,"say":"A binary prefix code is a set of nonoverlapping leaves in a binary tree. A codeword of length l occupies one part in two to the l of the tree's capacity.","live":[],"does":[[604.3088333333333,"heading_bound is shown on the screen, written out."],[614.3513333333333,"capacity_note is shown on the screen, written out."]]},{"start":616.0078333333333,"say":"Add those occupied fractions over all symbols. They cannot exceed one. This is the tree-capacity inequality, often called Kraft's inequality.","live":["capacity_note","heading_bound"],"does":[[616.3563333333333,"bound_work is shown on the screen, written out."],[617.4713333333333,"bound_work (the \"2^(-l_i)\" part) is emphasized."],[626.5268333333333,"bound_work (the \"2^(-l_i)\" part) is no longer emphasized."]]},{"start":627.1268333333333,"say":"Now compare the actual average length L with entropy H. Their difference can be written as this weighted logarithmic ratio.","live":null,"does":[[632.0263333333332,"bound_work is shown on the screen, written out."]]},{"start":636.2138333333332,"say":"The log-sum inequality says that weighted ratio is at least minus the log of the tree capacity. Since the capacity is at most one, that quantity is nonnegative.","live":null,"does":[[636.7013333333333,"bound_work is shown on the screen, written out."],[646.5233333333333,"bound_work is shown on the screen, written out."]]},{"start":648.3423333333333,"say":"Therefore every binary prefix code has average length at least H. Shorter paths for some outcomes consume more tree capacity and force compensating longer paths elsewhere. The average cannot cross the entropy bound.","live":null,"does":[[652.3133333333333,"bound_result is shown on the screen, written out."],[662.4483333333333,"A box is drawn around bound_result."],[663.2268333333333,"bound_result is hidden from the screen — left the board."],[663.2268333333333,"bound_work is hidden from the screen — left the board."],[663.2268333333333,"capacity_note is hidden from the screen — left the board."],[663.2268333333333,"heading_bound is hidden from the screen — left the board."]]},{"start":663.8268333333333,"say":"Entropy supplies ideal lengths such as one point three two two bits. Real codewords have whole-number lengths. Huffman chooses those integers optimally, and its average is always less than one bit above entropy for a memoryless symbol source.","live":[],"does":[[663.8268333333333,"huffman_bound is shown on the screen, written out."],[672.0693333333332,"huffman_bound (the \"H\" part) is indicated — a transient flash."],[675.9473333333333,"huffman_bound (the \"H + 1\" part) is indicated — a transient flash."]]},{"start":680.8893333333333,"say":"When probabilities are exact powers of one half, the ideal lengths are integers and Huffman can meet entropy exactly. Otherwise it rounds through the tree, as our small point zero seven eight gap demonstrates.","live":["huffman_bound"],"does":[[687.9593333333332,"huffman_bound is indicated — a transient flash."]]},{"start":695.4208333333333,"say":"There is also a way to make the rounding cost per symbol smaller. Encode blocks of symbols together. One possible extra bit is then shared by the whole block rather than charged to every symbol.","live":null,"does":[[700.4363333333333,"block_note is shown on the screen, written out."],[705.0923333333333,"block_note (the \"entire block\" part) is emphasized."],[708.1453333333333,"block_note is hidden from the screen — left the board."],[708.1453333333333,"huffman_bound is hidden from the screen — left the board."],[708.1453333333333,"block_note (the \"entire block\" part) is no longer emphasized."]]},{"start":708.7453333333333,"say":"Now return to files. Typical English text is structured. Spaces and common letters occur more often than rare punctuation, so even the one-byte distribution is uneven.","live":[],"does":[[708.7453333333333,"heading_files is shown on the screen, written out."],[711.7053333333333,"english_label is shown on the screen, written out."],[712.5993333333333,"english is shown on the screen, written out."],[720.2273333333333,"english is shown on the screen, written out."]]},{"start":721.6748333333333,"say":"English also has dependencies across positions. After the letters q and u, some next letters are far more plausible than others. Words, phrases, indentation, and repeated substrings create further predictability.","live":["english_label","heading_files"],"does":[[722.9403333333332,"english is shown on the screen, written out."],[735.6763333333333,"english is shown on the screen, written out."]]},{"start":737.4718333333333,"say":"A compressor turns that predictability into short descriptions. Huffman coding exploits a probability distribution. More advanced compressors also exploit context, repeated strings, and predictions from earlier data.","live":null,"does":[[740.0843333333332,"english is shown on the screen, written out."],[746.2373333333333,"english is shown on the screen, written out."]]},{"start":752.2588333333333,"say":"Independent random bytes are different. Each of the two hundred fifty-six byte values has probability one over two hundred fifty-six, and the next byte is not predicted by the bytes before it.","live":null,"does":[[753.2113333333333,"random_label is shown on the screen, written out."],[754.1393333333333,"random is shown on the screen, written out."],[758.0983333333332,"random is shown on the screen, written out."],[762.0343333333333,"random is shown on the screen, written out."]]},{"start":764.6668333333332,"say":"There is no statistical redundancy to exploit. The entropy is eight bits per byte, exactly the size already used by an ordinary byte.","live":["english_label","random_label","heading_files"],"does":[[766.2453333333333,"random is shown on the screen, written out."],[769.2763333333332,"random is shown on the screen, written out."],[771.3773333333332,"random is shown on the screen, written out."],[773.9898333333333,"english is hidden from the screen — left the board."],[773.9898333333333,"english_label is hidden from the screen — left the board."],[773.9898333333333,"heading_files is hidden from the screen — left the board."],[773.9898333333333,"random is hidden from the screen — left the board."],[773.9898333333333,"random_label is hidden from the screen — left the board."]]},{"start":774.5898333333332,"say":"There are two hundred fifty-six possible byte values, which is two to the eighth. If they are equally likely, identifying one requires eight bits.","live":[],"does":[[775.4253333333332,"random_count is shown on the screen, written out."],[783.2043333333332,"random_entropy is shown on the screen, written out."]]},{"start":784.8608333333333,"say":"For an n-byte file there are two to the eight n possible inputs. There are not enough shorter binary strings to give every input a unique shorter lossless description.","live":["random_count","random_entropy"],"does":[[790.0503333333332,"pigeonhole is shown on the screen, written out."]]},{"start":796.3163333333332,"say":"Some individual random-looking file may shrink by chance, but most will not, and compressor metadata can make them larger. Already compressed or encrypted files often behave similarly because their byte patterns have been deliberately made difficult to predict.","live":["random_count","random_entropy","pigeonhole"],"does":[[812.5703333333332,"pigeonhole is hidden from the screen — left the board."],[812.5703333333332,"random_count is hidden from the screen — left the board."],[812.5703333333332,"random_entropy is hidden from the screen — left the board."]]},{"start":813.1703333333332,"say":"The whole story now fits into four steps. First, a bit is one resolved binary choice.","live":[],"does":[[813.1703333333332,"heading_recap is shown on the screen, written out."],[816.5843333333332,"recap_one is shown on the screen, written out."]]},{"start":820.8293333333332,"say":"Second, repeated halving says an event of probability p carries about log base two of one over p bits. Averaging that surprise gives entropy.","live":["recap_one","heading_recap"],"does":[[821.1773333333333,"recap_two is shown on the screen, written out."]]},{"start":832.6913333333332,"say":"Third, Huffman coding repeatedly merges rare events so likely symbols end near the root. Its integer code lengths approach the entropy limit without ever beating that lower bound.","live":["recap_one","recap_two","heading_recap"],"does":[[833.1673333333333,"recap_three is shown on the screen, written out."]]},{"start":846.0388333333333,"say":"Finally, compression is possible when data contains predictable structure. English supplies redundancy and context. Independent random bytes do not. Entropy measures that difference, and coding turns the available predictability into fewer stored bits.","live":["recap_one","recap_two","recap_three","heading_recap"],"does":[[846.3873333333333,"recap_four is shown on the screen, written out."],[861.0393333333334,"A box is drawn around recap_four."],[863.6370625,"heading_recap is hidden from the screen — left the board."],[863.6370625,"recap_four is hidden from the screen — left the board."],[863.6370625,"recap_one is hidden from the screen — left the board."],[863.6370625,"recap_three is hidden from the screen — left the board."],[863.6370625,"recap_two is hidden from the screen — left the board."]]}]}]},"durationSeconds":865,"chapters":[{"title":"Questions Are Bits","startSeconds":0,"narration":"Suppose I secretly choose one of these eight letters. You may ask only questions whose answer is yes or no. Your job is to identify my letter with certainty. How many answers do you need? Because the letters are equally likely, the sensible first question cuts the possibilities into equal halves. Ask whether the letter lies in A through D. One answer leaves the upper four; the other leaves the lower four. The first answer has taken us from eight candidates to four. It has not named the symbol, but it has removed exactly half of the uncertainty. Ask another balanced question inside the surviving half. Four candidates become two. The same yes-or-no answer has again divided the remaining possibilities by two. A third question separates the final pair. Eight leaves now stand at the ends of the tree, and every letter is reached by exactly three answers. The count is not an accident. Three binary answers can form eight distinct answer strings, because two choices made three times give two cubed. Turn that relation around. The number of binary questions needed to choose among eight equal possibilities is log base two of eight, which is three. We give one balanced binary answer a unit of information and call that unit a bit. A bit is not merely a zero stored in memory. It is the distinction between two alternatives that were both still possible. With q binary answers there are two to the q possible answer strings. Therefore identifying one of N equally likely outcomes requires log base two of N bits. Next we will make the letters unequally likely. That one change will lead us first to entropy, the ideal average question count, and then to Huffman coding, a way to build an actual binary description close to that ideal."},{"title":"Likelihood Changes the Questions","startSeconds":119.25329166666668,"narration":"Keep the same eight symbols, but change their likelihoods. A now occurs forty percent of the time. B occurs twenty percent. C and D each occur ten percent, and the final four symbols each occur five percent. A alone occupies forty percent of the probability bar. It is twice as likely as B, and four times as likely as either C or D. C and D together occupy another twenty percent. They are less common individually, but they still matter much more often than any one of the four symbols at the narrow end. The final four slices are equal and rare. Each accounts for only one twentieth of the messages. If we keep the old balanced tree, every outcome still costs three questions. That treats a five-percent event exactly like the event that appears forty percent of the time. Instead, ask the most useful question first: is the symbol A? When the answer is yes, which happens forty percent of the time, the search ends after one bit. If it was not A, the next question can separate B. B then takes two answers. The commonest sixty percent of all symbols have received the two shortest paths. The price is that C and D now require four questions. That sounds worse than the old three, but each of those symbols appears only one time in ten. The four rarest outcomes take five questions each. Long answers are not free, but we pay for them on only twenty percent of the symbols. Now average the question counts using their probabilities. A contributes point four times one. B contributes point two times two. C and D together contribute point two times four, and the rare group contributes point two times five. The result is two point six questions per symbol. Some individual outcomes became more expensive, but the outcomes we see most often became much cheaper. The equal-depth plan cost exactly three questions. Exploiting likelihood saves point four of a question per symbol on average. Over a million symbols, that becomes roughly four hundred thousand fewer binary answers."},{"title":"From Question Counts to Entropy","startSeconds":255.6412916666667,"narration":"For eight equal outcomes, three halvings isolate one outcome because one eighth equals one over two cubed. Now replace one eighth by a general probability p. Imagine a very large collection of equally detailed possibilities. An event with probability p occupies the fraction p of them. To isolate that fraction by balanced binary questions, we keep halving. After q balanced questions, one answer path identifies about one part in two to the q. To match an event occupying fraction p, two to the q must be about one over p. Solve for q. The ideal question count is log base two of one over p. Equivalently, it is minus log base two of p. This quantity measures surprise. A likely event has a large p and a small information value. A rare event has a small p and needs a longer answer path to distinguish it. For A, p is point four, so the ideal length is about one point three two two bits. For a five-percent symbol, it is about four point three two two bits. These are not yet legal single-symbol code lengths. A single binary question cannot be split into a fraction. But across a long stream, common and rare symbols can share a description strategy, and a fractional average becomes meaningful. To find the ideal average, weight each symbol's question count by the probability of seeing that symbol, then add. This weighted average is the quantity we call entropy. Now calculate it for our distribution. A contributes about point five two nine bits per source symbol. That is its ideal length multiplied by the forty-percent frequency with which we pay it. B contributes about point four six four. C and D together contribute about point six six four. The four five-percent symbols each carry more surprise, and together they contribute about point eight six four bits. Adding every group gives an entropy of about two point five two two bits per symbol. Here is another way to read the same number. In a long block of n symbols, the probability pattern concentrates most messages among roughly two to the n H typical sequences. Distinguishing that many likely messages needs about the log of their count, which is n H binary answers. Entropy is therefore not an arbitrary formula. It is the question count forced by the number of likely messages."},{"title":"Build the Huffman Tree","startSeconds":417.7835625,"narration":"Huffman coding turns the probability list into a binary tree. Begin with one leaf for every symbol. The only repeated instruction is to merge the two smallest weights. The first two rarest leaves are E and F, each with weight five. Join them under a new node of weight ten. G and H are the next equal pair. Merge their five-percent weights into a second node of weight ten. C and D each have weight ten. Merge those two leaves into a node of weight twenty. The two ten-weight branches made from E through H are now the smallest pair. Their merge produces another weight twenty. Three weights of twenty are available. Ties may be resolved consistently in more than one way. Merge these two compound branches to make forty. B has weight twenty and the compound branch has weight forty. They are the two smallest remaining items, so their parent has weight sixty. Finally merge A, with weight forty, and the remaining weight sixty. Their root has weight one hundred, and the Huffman tree is complete. Assign zero to the upper branch and one to the lower branch at each split. A codeword is the sequence encountered while walking from the root to a leaf. A is reached immediately by code zero, so the most likely symbol costs only one bit. B follows one then zero, so it costs two bits. C and D sit deeper, with codes one one zero zero and one one zero one. Each requires four bits. The four rarest symbols lie one level deeper still. Their codes share the prefix one one one, then use two more bits to select the leaf. No codeword is the beginning of another codeword. That prefix property lets a decoder read the stream from left to right and know exactly when each symbol ends. Compute the average length. A pays one bit forty percent of the time. B pays two bits twenty percent of the time. C and D pay four, and the final four symbols pay five. The Huffman code therefore uses two point six bits per symbol on average. The merging rule has recovered the likelihood-aware question plan, and it has guaranteed that no other binary prefix code has a smaller average length for these individual symbols."},{"title":"The Bound and the Files We Compress","startSeconds":569.1103333333333,"narration":"Put the two numbers on one scale. Entropy for our source is about two point five two two bits per symbol. The Huffman code uses two point six. The gap is only point zero seven eight of a bit per symbol. Huffman coding is not merely better than the fixed three-bit code. It is very close to the information-theoretic target. Two questions remain. Why can no lossless prefix code move the green point to the left of entropy? And if the bound is unbeatable, why is Huffman allowed to sit slightly above it? A binary prefix code is a set of nonoverlapping leaves in a binary tree. A codeword of length l occupies one part in two to the l of the tree's capacity. Add those occupied fractions over all symbols. They cannot exceed one. This is the tree-capacity inequality, often called Kraft's inequality. Now compare the actual average length L with entropy H. Their difference can be written as this weighted logarithmic ratio. The log-sum inequality says that weighted ratio is at least minus the log of the tree capacity. Since the capacity is at most one, that quantity is nonnegative. Therefore every binary prefix code has average length at least H. Shorter paths for some outcomes consume more tree capacity and force compensating longer paths elsewhere. The average cannot cross the entropy bound. Entropy supplies ideal lengths such as one point three two two bits. Real codewords have whole-number lengths. Huffman chooses those integers optimally, and its average is always less than one bit above entropy for a memoryless symbol source. When probabilities are exact powers of one half, the ideal lengths are integers and Huffman can meet entropy exactly. Otherwise it rounds through the tree, as our small point zero seven eight gap demonstrates. There is also a way to make the rounding cost per symbol smaller. Encode blocks of symbols together. One possible extra bit is then shared by the whole block rather than charged to every symbol. Now return to files. Typical English text is structured. Spaces and common letters occur more often than rare punctuation, so even the one-byte distribution is uneven. English also has dependencies across positions. After the letters q and u, some next letters are far more plausible than others. Words, phrases, indentation, and repeated substrings create further predictability. A compressor turns that predictability into short descriptions. Huffman coding exploits a probability distribution. More advanced compressors also exploit context, repeated strings, and predictions from earlier data. Independent random bytes are different. Each of the two hundred fifty-six byte values has probability one over two hundred fifty-six, and the next byte is not predicted by the bytes before it. There is no statistical redundancy to exploit. The entropy is eight bits per byte, exactly the size already used by an ordinary byte. There are two hundred fifty-six possible byte values, which is two to the eighth. If they are equally likely, identifying one requires eight bits. For an n-byte file there are two to the eight n possible inputs. There are not enough shorter binary strings to give every input a unique shorter lossless description. Some individual random-looking file may shrink by chance, but most will not, and compressor metadata can make them larger. Already compressed or encrypted files often behave similarly because their byte patterns have been deliberately made difficult to predict. The whole story now fits into four steps. First, a bit is one resolved binary choice. Second, repeated halving says an event of probability p carries about log base two of one over p bits. Averaging that surprise gives entropy. Third, Huffman coding repeatedly merges rare events so likely symbols end near the root. Its integer code lengths approach the entropy limit without ever beating that lower bound. Finally, compression is possible when data contains predictable structure. English supplies redundancy and context. Independent random bytes do not. Entropy measures that difference, and coding turns the available predictability into fewer stored bits."}]}}