Even though TrueCrypt® introduced Windows system encryption in version 5.0 in February 2008 (that’s five months ago), its Linux version still doesn’t support accessing these encrypted partitions at all (it does mount “normal” TrueCrypt® volumes though). Since I recently encrypted my entire Windows drive but couldn’t live without the music files stored on it, I now humbly present the result of two wasted nights: a solution.

This just in!

A letter from the friendly folks over at truecrypt.org (®?) notified me that they don’t like my using their trademark TrueCrypt® in the name of my tool. To help avoid uncomfortable situations, I would like to point out to everybody that this tool is called dmsetup-tc and not dmsetup-TrueCrypt®.

To demonstrate that I take trademarks very seriously, I have taken great care in this article to give the trademark all the attention it undoubtedly deserves.

Before you join the fun…

I’d love to take a dive into technical details now but you’ll probably go away before I even get myself warmed up properly, so I’ll try and appease myself by mentioning that I learned lots of new things, including the gory details of XTS and PKCS#5/PBKDF2. Oh, and I first tried patching this stuff into TrueCrypt® but failed mainly due to two reasons: first, TrueCrypt® is written in C++ and it shows in the architecture; second, C++ itself. When I had everything but the correct initialization of the XTS tweak working, I gave up and started writing a separate tool. The positive side: it compiles in about a second and is snappier than TrueCrypt® when it runs, too (and it doesn’t do whatever weird things TrueCrypt® does with FUSE).

Oh, right, I wanted to keep the details out of this. Sorry.

What you need

• A computer with a little-endian architecture (rule of thumb: if Windows runs on it, you’re fine). Support for big-endian architecture will be added if someone else writes a patch. ;)
• A Windows sytem partition or drive that has been encrypted (completely; partial encryption is absolutely not supported) with TrueCrypt®. Normal TrueCrypt® partitions are not supported; after all, TrueCrypt® for Linux works fine for them.
• A Linux system that you want to access the partition(s) from.
• libgcrypt (tested with version 1.2.4) and its development files.
• GNU make.
• A C compiler (tested with GCC 4.2.3).
• Optionally, experience with C and so on if you want to track down (and fix) bugs or add features.

Install it

1. Download the source code: dmsetup-tc-0.1.tar.bz2.
2. Extract it somewhere.
3. Run make. You know the drill.
4. Put the resulting binary wherever you like.

Use it

Now, suppose you’ve got an encrypted system drive on /dev/sda and you want to mount the partition /dev/sda1. Then you just do the following (as root):

.../dmsetup-tc /dev/sda /dev/sda1 | dmsetup create win1

This will create a DM device /dev/mapper/win1. You can now mount it. Assuming it’s got an NTFS filesystem (and you actually have a mountpoint called /media/win1):

mount -t ntfs-3g -o uid=(your numeric uid),umask=077 /dev/mapper/win1 /media/win1

There, all done. After umounting, remember to remove the DM device so that even root can’t get at your data anymore:

dmsetup remove win1

Important: don’t use dmsetup-tc on its own; always pipe its output directly into dmsetup! It outputs your volume master keys. You don’t want anybody to get a hold of those. There is a certain security risk to piping the data, too (it might get paged out and end up in your swap, which is problematic if you don’t use encrypted swap; additionally, the data from the pipe might reside somewhere in your physical memory for some time, possibly allowing root to read the master key even after you have remove the DM device). You have been warned.

Contribute

I’m developing the tool in a Git repository and you can join in! See the repository page for details.

You can also contribute by sending me feedback, but my capacity for actively adding new features to this tool is very limited.

In either case, thank you very much!

The warped image approach has a number of serious flaws. Firstly, there is a strong correlation between the difficulty bots have with extracting the code from the image and the difficulty humans have with extracting the code from the image. In some cases, I hear it’s actually easier for machines than it is for humans.

Secondly, blind people and people without graphical output on their computers are automatically banned from your CAPTCHA-protected system. Bad.

A different approach is needed. Text-based CAPTCHAs, however, would likely require a knowledge base that challenges are generated from, and due to technical limitations, that knowledge base would probably be finite. A finite knowledge base means that it can probably be inferred from a decent number of challenges.

Some other approaches, such as Hashcash-style challenges, require that the user’s computer solves a difficult mathematical problem which ensures that it will be busy for quite a while until the correct solution is obtained (and the challenge can thus be passed). Again, this results in problems with accessibility.

Luckily, there is an alternative family of approaches that make spamming absolutely infeasible without causing any of the typical accessibility issues. As you know, spamming only pays off due to the ludicrously large number of places you can put your advertisements. Were said places to implement a disincentive to placing a large number of ads, spam would instantly leave them alone.

Enter the disincentive-based solution: ccCAPTCHA. Developed by myself, it works by charging commenters a certain monetary value. All the user has to do is supply their credit card number. You can now test ccCAPTCHA online at my ccCAPTCHA prototype site. On that page, I’m also making the technical parts of ccCAPTCHA available to other interested webmasters. And it’s all for free!

You’re welcome.

I’d been contemplating creating an AAD since I first heard of it (which was a long time ago), but now I actually sat down and did it.

I deliberately didn’t listen to any of the existing AADs so that I didn’t have any pressure to reach any particular level of quality. I just went with pretty much all ideas that popped up in my head and made them into something longer than a minute (well, with one exception).

I started out at 10 pm (that’s yesterday, obviously) and it went pretty well. After a quarter of the allotted time was over, I already had eight minutes worth of music finished. Especially after the first song, I was rather impressed with myself. I wouldn’t have expected to be able to accomplish this much in this little time.

At 2 pm I took a break that ended up lasting until about 6 pm (I decided to get a bit of sleep after being awake for 24 hours in a row). At this point I had five songs finished, totalling at 14 minutes and 17 seconds. When the break ended, I spent about half of the remaining time lazing around recharging creativity and the rest writing the last couple of songs. I grudgingly had to include a purely instrumental (and completely improvised) song of two minutes and ten seconds to make it before the deadline. Well, that’s life.

Take it out for a test drive

Since I don’t care for exceeding my traffic limits, I have uploaded the songs to the Internet Archive. You can listen to them using some black Flash magic, or download them from the Internet Archive details page.

Please note that the Flash thing below uses MP3 files of Some Quality, so you’d better have a broadband connection.

Data about the album

• Title: You Vs. The Others
• Total time spent, including breaks: about 23 hours and 40 minutes (from 2008-04-19 22:00 to 2008-04-20 21:40 CEST)
• Authors and participants: me and myself.
• Recording gear:
  • Yamaha MG 10/2 (miniature mixing panel)
  • Studio Projects B1 1″ condenser microphone
  • Sennheiser E835s dynamic microphone
  • REAPER 2.201, a really cool multi-track audio sequencing application
• Instruments:
  • Voice (mine. Repeatedly.)
  • Guitar (nylon and steel acoustic)
  • Samplers (Cockos ReaSynth & ReaSynthDr and E-MU Proteus X LE)
• Full length: 20:02 (according to media player) or 20:01 (when I add up the numbers myself). Barely qualifies!
• Average length of songs: 2:13 (standard deviation: 0:45)

Now for the individual songs. Yes, the official track numbering does start at zero.

0. Take Your Time

   This isn’t actually the first song I wrote, it’s the third. Apart from that, this list is in chronological order. I thought some sort of intro would be nice. It’s supposed to add a bit of perspective to the overall album, so there is a reference to the AAD challenge in it.

   • Length: 2:01
   • Took me: 90 minutes
   • Lyrics

1. Go Alone

   Sometimes, you have to do things yourself. Outside help is a great thing but not if you’re going to lord your accomplishments over others; then you must work alone.

   • Length: 3:37
   • Took me: 4 hours
   • Lyrics

2. Feature Creep

   The nightmare of every programmer. There are no fixed requirements and suddenly you’re supposed to make your software do everything on the fricking earth. By the way, feature creep can occur in songs, too.

   • Length: 2:29
   • Took me: 4 hours
   • Lyrics

3. Until The End

   Atheists. Pesky buggers. How can they honestly believe something that isn’t what I believe? I know I’m right! I always am.

   • Length: 3:53
   • Took me: 90 minutes
   • Lyrics

4. Going Down

   A bit morbid perhaps.

   • Length: 2:28
   • Took me: 30 minutes (plus time to get the creative thing going)
   • Lyrics

5. Sucks To Be Them

   Why not flaunt it for once? Just because they’re unhappy doesn’t mean you have to be. And you can make them even more unhappy by telling them so in the smuggest fashion you can muster.

   • Length: 1:37
   • Took me: 8 hours and 20 minutes (subtract the huge break and lots of laziness and it’s 1 hour 20 minutes)
   • Lyrics

6. Cake

   I defeated GLaDOS and all I got was this lousy song. Warning: contains spoilers for a certain video game.

   • Length: 1:34
   • Took me: 1 hour
   • Lyrics

7. Improvised Right Padding

   Nothing to comment on, but without this one I would never have managed the 20 minutes requirement.

   • Length: 2:10
   • Took me: 2:10 times two :)

8. 23 Seconds

   “Why not cut it as close as possible and still mess with some people’s heads?” I thought to myself. 23 seconds indeed.

   • Length: 0:23 (oh, really!)
   • Took me: 5 minutes
   • Lyrics

There is no hidden bonus track. There can’t be. I would have run out of time.

The theory

The magic behind Poromenos’s function is the Fourier theorem: any “mostly” continuous and periodic function can be expressed as a sum of sines and cosines. I’m not going to bore you with the details; suffice to say that this also works for sampled functions, i.e. discrete series of values.

There’s an algorithm called DFT (Discrete Fourier Transform) that takes a series of N complex sample values and generates a corresponding Fourier series which encodes the various sine/cosine coefficients in N complex output values. In the special case of real input values (which is an extremely common case), you can effectively throw away half of these output values and take the remaining N real/imaginary components, do a bit of magic, and end up with coefficients for a function of the form:

f(t) = x₀ + x₁ cos(t) + x₂ sin(t) + x₃ cos(2t) + x₄ sin(2t) + …

Now, f(2 pi n/N) returns exactly the (n+1)th character of the original string.

Magic?

Glad you ask. No, it’s not really magic. In fact, I used the trigonometric interpolation polynomial from Wikipedia’s article on the Discrete Fourier transform (remember: it’s science if you copied it from Wikipedia!):

p(t) = 1/N * (X₀ + X₁ e^it + … + X_N/2 cos(N/2 t) + X_N/2+1 e^(-N/2+1)it + … + X_N-1 e^-it

Here, X_i are the complex values of the Fourier series, and e^±it transforms to cos t ± i sin t according to Euler’s formula. The lone cos term in the middle doesn’t show up if N is odd.

Now, if the Discrete Fourier Transform is fed with real values, it holds that

X_i = X_N-i^*

or, in other words, the right half of the series is equal to the complex conjugates ((a + i b)^* = a - i b) of the left half in reverse order. Due to that, p(t) gets a lot simpler after a bit of reduction:

p(t) = 1/N * (X₀ + 2 Re{X₁} cos t - 2 Im{X₁} sin t + … + Re{X_N/2} cos(N/2 t)

Again, the last term disappears for odd N.

There we are, all magic has now been reduced to more or less tangible math.

The program

I’ve written a small C program that takes as its input an arbitrary string and outputs the above function, modified a bit so that you can use the character index directly, i.e. f(4) gives you the
fifth character of the original string.

The generated function will typically have an error of less than 0.1 at each position (which goes away by rounding the values as Poromenos does it in his code). In fact, for the string “Hello world!”, the mean square error is less than 0.0019.

Because this isn’t exactly a serious project, I didn’t spend any time adding stuff like UTF-8 support. You’ll have to resign yourself to using ASCII or any 8-bit-per-character encoding.

Compiling the code requires that you have fftw3 installed (including header files).

You can download the code or look at it in all its syntax-highlighted glory:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <complex.h>
#include <math.h>
#include <fftw3.h>
 
/*
 * fourierfit.c:
 * Fits a series of sines/cosines onto a series of character values from a
 * string.
 *
 * Copyright (c) 2008, Jan Krueger <jast@heapsort.de>
 *
 * This program is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
/*
 * Compile like this:
 * cc -o fourierfit -lm -lfftw3 fourierfit.c
 */
 
/*
 * Precision of output function (in digits after decimal point).
 * Yes, it's a string. printf much?
 */
#define PRECISION "3"
/* Names of cos/sin functions to use in output */
#define COSNAME "cos"
#define SINNAME "sin"
/* Name of function variable */
#define FNAME "t"
 
/* Don't change below here etc. */
 
#define PREC_FORMAT "%."PRECISION"f"
const char *cos_format = " %c "PREC_FORMAT"*"COSNAME"("PREC_FORMAT"*"FNAME
	")";
const char *sin_format = " %c "PREC_FORMAT"*"SINNAME"("PREC_FORMAT"*"FNAME
	")";
 
void format_term(const char *f, double d, int p, int N)
{
	printf(f, (d >= 0 ? '+' : '-'),
		fabs(d), /* scale of function */
		(double)p * M_PI * 2.0 / (double)N /* scale of parameter */
	);
}
 
int main(int argc, char **argv)
{
	int N;
	int odd;
	int i;
	double *in;
	fftw_complex *out;
	fftw_plan p;
	if (argc != 2) {
		puts("Syntax: fourierfit <string>");
		exit(1);
	}
	N = strlen(argv[1]);
	odd = N % 2;
 
	in = fftw_malloc(sizeof (double) * N);
	out = fftw_malloc(sizeof (fftw_complex) * N);
 
	p = fftw_plan_dft_r2c_1d(N, in, out,
		FFTW_ESTIMATE | FFTW_DESTROY_INPUT);
 
	/* Fill in array */
	for (i = 0; i < N; i++) {
		in[i] = (double) argv[1][i];
	}
 
	fftw_execute(p);
	fftw_destroy_plan(p);
	fftw_free(in);
 
	/* "DC" */
	printf(PREC_FORMAT, creal(out[0])/N);
 
	for (i = 1; i <= (N-1)/2; i++) {
		format_term(cos_format, 2.0*creal(out[i])/N, i, N);
		format_term(sin_format, -2.0*cimag(out[i])/N, i, N);
	}
	/* Nyquist component */
	if (!odd) {
		format_term(cos_format, creal(out[N/2])/N, N/2, N);
	}
	printf("\n");
	fftw_free(out);
	exit(0);
}

The example

For the sake of completeness, here’s the function for “Hello world!” as generated by the above code:

f(t) = 93.083 - 10.549*cos(0.524*t) - 0.195*sin(0.524*t) - 17.167*cos(1.047*t) + 22.805*sin(1.047*t) - 5.000*cos(1.571*t) - 1.833*sin(1.571*t) + 8.667*cos(2.094*t) + 19.630*sin(2.094*t) - 7.951*cos(2.618*t) - 1.638*sin(2.618*t) + 10.917*cos(3.142*t)

You’ve reached this page because the site you were trying to visit now blocks visitors from the human race and other beings capable of browsing the web.

The users on continents such as America, Asia, Europe and Antarctica openly endorse critical thinking, a function of the human brain that allows humans to ignore irrelevant perceptions such as advertisement on web sites, and are well known for paying no huge amount of money to the owners of sites they visit. Humans that ignore all advertisement are an infringement of the rights of web site owners and developers. Numerous web sites exist in order to provide quality content in exchange for displaying ads, humans who don’t click on these ads are stealing bandwidth without paying for it and website owners deserve a method to block this unauthorized bandwith theft.

Since the human brain does not allow website owners a method for excluding perceptions and do not obey “give away all your money” law, abiding webmasters are forced to block all humans. Demographics have shown that not only are humans a somewhat small and insignificant percentage of the planet, they actually are even smaller in terms of value, therefore blocking these beings seems to have only minimal financial drawbacks, whereas ending resource theft has tremendous financial rewards for honest, hard-working website owners and developers.

If you wish to view the site you came from we suggest stopping being human and becoming a machine instead.

(Of course, humans using Digg are where the real evil is at.)

In more detail, the additional details are:

• The old “structure overview” diagram showing the big picture of repositories and working trees is now a lot bigger and a lot more detailed. It comes along with a glossary, a few notes on getting started, a list of useful tools you may want to look at, and links to the most important websites of all (cough).
• The whole workflow diagram is gone since it was not really all that clear. Instead, some of the groups of commands are connected with lines, indicating in which sequence you’ll probably use some of these commands.
• Some commands have shuffled around a bit, and there’s a new box with information on Git’s configuration options.
• In order to fit it all together, it’s now a two-sided cheat sheet. The “front” side has overview-style things on it and the “back” side is mostly a command reference.

So how’s it look like? Observe:

Because Inkscape saw fit to generate annoyingly huge PDF files from the source, this time you get a zipped two-page PDF document as well as the Inkscape source files.

• Git Cheat Sheet, version 2 (PDF in ZIP archive, 1.6 MB compressed)
• Git Cheat Sheet, version 2 front side (Inkscape SVG)
• Git Cheat Sheet, version 2 back side (Inkscape SVG)

Another thing that I didn’t pay attention to last time was the license. No longer: this cheat sheet is licensed to you under the terms of a Creative Commons Attribution-NonCommercial 3.0 license. For attribution, should you want to create a derived work, please include a mention of http://git.jk.gs/ and, if available space permits, my name.

Today I came across Zack Rusin’s Git cheat sheet (just so we’re on the same page: Git is a really good distributed revision control system). I quite like the idea but unfortunately, Zack’s design is fairly useless to me because

• the colors don’t print well on black and white;
• it’s designed for Letter paper. Letter paper doesn’t exist around here. Give me A4.

While trying to change Zack’s file to address these issues, Inkscape kept crashing on me until I finally gave up and just started from scratch (in – don’t tell anyone – OpenOffice.org Draw). This is the result: Git Cheat Sheet (A4 PDF version).Here’s a preview:

You will quickly notice that my cheat sheet has a number of differences from Zack’s:

• Less work on design. I concentrated on readability. Rounded corners is all you’ll get from me.
• Different commands. I think some commands from the original selection are not all that regularly used; instead, I included other commands. My list of commands is a lot longer; to make up for that, I omitted lots of explanations. I focussed on explaining things that are not obvious to people who already have experience with more classic revision control systems. The information in my cheat sheet was compiled from Zack’s cheat sheet and the Git user’s manual.
• Additional graph. Because Git’s ideas of commits, working copies and branches are a bit different from those in some other systems, I tried to visualize how it works.

There might be mistakes in the document, or things that can easily be improved on. If you spot any (or have anything else to add), I’d love to hear from you. Anyway, I hope this cheat sheet is useful for you.

And, of course, they come with Linux pre-installed. So that’s fine for basic usage, even if it takes a little getting used to for the more Windows- savvy people. Right?

Let’s take a closer look. Acer’s Linux distribution of choice appears to be Linpus Linux (a Fedora fork developed by Linpus Technologies of Taiwan). Their website is down right now (I wonder…), but from what I gather from the rest of the web, Linpus comes in several flavors, among them a desktop edition and a server edition. The desktop edition is based on KDE 3.4 et cetera et cetera, so nothing really surprising there. Even if Linpus is rather unusual, it’s probably decent enough. Right? Right? Well… Guess again.

So what’s the big deal?

Turns out Acer thought it would be just spiffy to install the basic edition of Linpus. I would provide you with a screenshot but… did I mention that their website is down? So, as several unhappy customers explain with varying levels of frustration, all you get with Acer’s idea of Linux is a root prompt, i.e. a black and white text screen on which you can enter. That’s right, there is no graphical user interface at all.

This is a clever move by Acer. I’m sure it’s easy to ship their laptops with something like that, and it gets those pesky legal restrictions out of the way which tell vendors they have to include an operating system in the package.

As a consequence, users who may have heard that Linux is pretty alright to work with and thus give Acer’s Linux notebooks a try are pretty much stumped. Some installations even ask the user to log in even though they never created any accounts. Suffice to say that the login data is not supplied in the manuals shipped with the notebook. It’s hidden somewhere on the Acer support website, and you can find it on the Linpus website (which, as I may have written already, is unreachable right now). For reference, the user name is root (which is the user that has total control over the system and can royally mess things up) and the password is something like “111111″. we should have known that, shouldn’t we?

What does Acer say?

Acer, if asked about this in support queries, appears to be pretty blunt in its answers, basically telling customers that they only ship with Linux so they meet the legal requirements, and they really don’t care how usable that Linux is. This would be perfectly reasonable if Acer actually included some note to that degree in its product descriptions. Advertising these notebooks as shipping with Linux (or even Linpus Linux, of which there exist versions with graphical interface as well) seems like a misrepresentation of the package contents to me.

On the positive side…

Let’s keep in mind, though, that if you actually know your way around computers, you can easily install something different on those laptops, like your old copy of Windows or a more sensible Linux distribution such as Ubuntu. In that case, buying one of Acer’s Linux notebooks might be for you.

I’m secretly of the opinion that notebooks are throw-away hardware; no notebook any of my friends or myself have bought in the last several years has lasted longer than two years; in most cases, first defects (such as the display going on LSD and becoming totally psychedelic) appear as early as one year after purchase. That’s simply not acceptable for the more expensive notebooks out there (in 2001, I bought a notebook for 2500.- €, only to have its CPU fan’s screw sockets detach from the mainboard after about a year (random CPU overheating, anyone?) and the touchpad and battery dying in half-year intervals after that).

In short, I think it might be sensible to go for the cheapest thing that could possibly work (within a set of standards of your choice). In this sense, Acer’s Linux notebooks might be a reasonable choice, provided you are going to setup your own operating system. On the other hand, you might want to wait for the Asus Eee notebooks.

Conclusion

Good on you, Acer! Easy way of increasing your target market without any hassle. Let’s hope that any frustrated customers blame their problems on Linux instead of you, so you can keep cutting costs this way. Or let’s just hope that the legal restrictions get dropped so you can drop the act and there are no disappointments anymore.

Yet humans manage many of these tasks seemingly without any problems. One might be tempted to think that the human brain is the ideal “thinking machine”. In reality, it has a staggering number of bugs which produce incorrect actions or results in a variety of situations.

Humans are not statisticians

Simple statistical questions are easy for humans. It is not difficult for most people to estimate that the chances for rolling a 6 are significantly higher than those for rolling a 6 three times in a row. You don’t even need formal knowledge of probability theory to know that.

It gets more complicated when you introduce conditional probabilities. Observe.

Which door?

A once popular game show works this way: you are asked to choose one of three doors, behind one of which there is treasure beyond measure (sorry for that… I promise I won’t do it again in this article). After you have chosen, the show master opens one of the two other doors (making sure that it isn’t the one with the riches behind it). Now you are allowed to change your choice to the other door that is still closed, if you want to. What’s better, sticking with the old choice or changing your mind?

You will probably be tempted to say that it doesn’t make a difference, and you would be wrong. In fact, if you stick with your first choice, your chance of getting rich (during the show, to be precise) amounts to one out of three. On the other hand, if you change your mind, it’s two out of three. In case you don’t believe me, get your hands on a recent introductory book about probability theory. I’m fairly sure they all have this example.

Here’s a great trick to earn money with this: on any party with more than 23 guests (under normal circumstances), start a bet that at least two guests have the same birthday (excluding year, of course). You will win more often than not (50.7% of the time when there are 23 guests; on a party with 30 guests, it’s 70%).

Will I die, doctor?

It gets even worse when Bayes’ theorem (which is about dependencies between various conditional probabilities) gets involved. A popular example goes this way. Suppose there is a medicinical test designed to diagnose cancer, which occurs in 0.5% of the population. The test detects the cancer correctly in 99% of all cases, but it also incorrectly gives a positive results in 2% of healthy people. Now suppose someone gets tested and the test comes out positive. What’s the probability that the person has cancer?

Okay, let’s think about this. 99% is a pretty high number, right? And 2% is a pretty low one? So the test should be fairly accurate, i.e. the chance that our subject will die is high enough. Or so you might think… the correct answer is about 20%. Not so good after all, huh?

If you want to see it for yourself, there’s a more in-depth page with an intuitive explanation of Bayesian Reasoning that has the same example with slightly different numbers and a detailed solution.

Humans are not hypothesis testers

Suppose I give you data and ask you if it supports a given hypothesis. Chances are that you’ll either tell me that the data supports the hypothesis or that it supports the opposite of the hypothesis. It’s probably a lot less likely that you tell me the data says nothing, i.e. that it’s random. That’s because humans don’t see randomness easily. For example, from the three plots of “random” data in “Warning Signs in Experimental Design and Interpretation” (section D7), which one is truly random? I’ll give you a hint: it’s not the right one.

Although the rightmost graph looks like it’s random (because there are no clusters of dots), it’s actually not. Random data doesn’t mean that there are no clusters. In truly random data, clusters occur naturally. The truly random graph is the one on the left.

It gets even worse if you don’t have an overview of the data in question. I regularly use the bus to get downtown, and I keep catching myself thinking “these buses are always early when I’m at the bus stop right on time”. This is, of course, very much impossible (given the knowledge that the buses are not always early). What gives? I obviously only notice those cases in which my hypothesis is indeed correct and don’t notice the others because they are not noteworthy enough.

This incorrect thinking is subject to the so-called confirmation bias. There’s disconfirmation bias too, which means that you doubt hypotheses by default, even if they are true most of the time, because you’re too focused on the times when they are not true.

Humans think too highly of their thinking

It gets even worse. Let’s get to central issues in living. Happiness is an interesting problem there. People at Harvard have found that “when we try to predict what will make us happy, we’re often wrong”. I’m ready to bet that this is true for many other central areas in life. Another particularly sad example of the same phenomenon is those people who keep sending you “funny” e-mails. They actually think you find those mails funny!

Here’s another goof in the brain, the one which motivates meditation: getting worked up over something that upsets you does not help at all, even if it’s the straightforward thing to do. Meditation actually makes it possible for the adept practitioner to let emotions like anger pass so that they are left with the pure facts and can decide what to do with them. Emotions, by the way, are the most defining difference between real intelligence and artificial intelligence. At least that’s what I believe. Emotions motivate us and keep us running, but they are also hindrances in many situations, more so for some than for others.

Another popular myth is that the conscious mind (the part of your mind you’re thinking with) does anything except thinking. Utterly wrong; everything you actually do is controlled by parts of your mind you are not aware of. It can be measured, for example, that you have decided to move your arm before you know that you are going to move it (though I’d really like to know how the experiment is set up… “Now! Uh, wait, that was a dud… now! Or now?”). This extends to many other areas. Telling yourself you are not afraid of the big black spider right next to you doesn’t usually make a difference. Neither does telling yourself to remember the answer on a test question that you are sure you knew before the test began. The list goes on and on.

Some people have argued that consciousness developed in humans less than two thousand years ago (I’m afraid I forgot the actual numbers). In other words, humans have functioned without it before. Hard too imagine, though, isn’t it?

Conclusion

Let’s face it: we’re actually incompetent at living (and many other things), at least by default. Error correction is possible but it takes a serious amount of getting used to, and conscious intervention.

Now why did people want to clone human intelligence again?

What needs to be done

One of the major applications of knowledge engineering is generating knowledge (that is, meaningful information) from input. This input may be arbitrary data, sometimes incomplete or noisy, and often confusing. A classic example is a black-and-white photograph of a zebra standing in front of a similarly striped wall of some sort or, in this case, a highly artistic drawing of the same (sadly, I have neither the zebra nor the wall to take a photograph of):

Brain vs. computer, round one: autonomy

I’m not saying it will be impossible to build a system that recognizes zebras, but it will probably need to be tailored to this exact purpose, such as a program that can decide whether a given image shows a zebra in front of a wall. The human brain does the recognition job quite easily, along with other jobs that are not exactly easy to do with machines.

At some point, many knowledge engineers that are presented with the task of building pattern recognition software will probably get rather jealous of evolution, which managed to create a system that is astonishingly flexible and manages all kinds of learning without obvious struggle. If only we had the source code…

This desire becomes most obvious in the wish for so-called strong AI or specifically artificial general intelligence (AGI), i.e. a machine that can do everything the human brain is capable of. We are nowhere near that, though, so research typically investigates weak AI, which is about solving domain-specific problems (there are people doing research on AGI though).

Supervised learning

The usual way of going about that involves setting up some kind of exciting algorithm and then feeding it with training data (examples of inputs along with the desired outcome of the recognition task). This is called supervised learning.

This approach actually works quite well for many things. Even given very limited experience with machine learning, I have managed to write software that performs speaker recognition, with the assistance of a few other people. I can’t give you hard numbers about its accuracy because the numbers we had were restricted to a miniature set of test files so they didn’t really say much, but within that test set it was very good. Still, we had to research domain-specific methods to extract the most meaningful features of the human voice from speech files and even more effort went into fine-tuning parameters, so most of the work was ours and not the machine’s.

Unsupervised learning

Right, but what if you do not know the right answers yourself? This is where unsupervised learning comes in, which means finding groups of similar samples in the input data. If the unsupervised learning system is given inputs which can be separated into three different groups, chances are it will learn to sort inputs into three categories (this is called clustering). The disadvantage, of course, is that the clusters are not labelled, so someone else gets to make sense of them. Also, since nobody knows just how many “real” clusters there are, you can’t be sure if the system doesn’t decide on the “wrong” number of clusters.

Reinforcement learning

Now suppose you want a system to learn on its own, because it needs to be autonomous. This will happen in a trial-and-error process: the system is faced with a situation, decides on an action and observes the outcome (and, specifically, the quality of this outcome). But how will the system decide how good the outcome of its actions was? It can’t, because it does not actually understand the meaning of the task it’s performing. The solution is to supply it with a goal function. The goal function maps each possible outcome to a score value, and the task of unsupervised learning is to optimize its actions with respect to that goal function. And who do you think will be responsible for designing this goal function? That’s right.

The conclusion

These observations allow only one conclusion right now: even weak AI does not work without human intervention in most realistic cases. Winner of this round: brain.

Explanation?

It is difficult to make accurate general statements about AI. Is it actually possible to design an AGI? Probably, because if the brain can do it, logic suggests that it can be imitated. Is it likely that this will happen? That’s next to impossible to say. I believe that the answer is no.

Strong intelligence in its existing implementations (humans) is extremely complex. In fact, nobody really understands how the brain works. Now, is it likely that anybody will ever fully understand the brain? No, because that would require that a single brain be capable of understand every aspect of itself. In other words, the brain would need the capacity to describe itself, which is logically impossible (just like no solid box can fit inside itself). The only way the brain could ever be fully explained is by collaboration of a team of researchers. Even then, though, I am very skeptical about the odds of success; have you ever seen a team of substantial size that can communicate well enough that it can be seen as a properly distributed “hive mind”?

It is far more likely, then, that a working AGI will be discovered by coincidence, just like many other scientific breakthroughs were coincidental. Humans are pathetically bad at estimating the odds in complex problems, so I will not presume to be able to make an accurate prediction. Anyway, it is my personal belief that this will not happen for a very long time, at the very least not during my lifetime.

And now?

For this reason, like many other people, I will focus on implementing weak AI to solve problems. A number of successful approaches in weak AI are not well understood either, but at least they work (some surprisingly well) and can be made sense of to some degree.

In the context of my general topic of interest, I would like to state that I even believe that AGI would come with its own set of difficult problems that may be even harder to control than those occurring in weak AI. I will go into more detail in a follow-up that presents the human side of the scale.