Stop the Autoconf Insanity!
Why we need a new build system
Any veteran GNU/Linux user has at one point or another run across a package that used the autoconf/automake tool-set. There is a lot to be said about this emerging standard. Running './configure', 'make', 'make install', usually results in a working installation of whatever package you are attempting to compile. The Autoconf tools are also portable to almost every *nix platform in existence, which generally makes it easier to release your program for a large variety of systems. However, despite these few pluses, the auto* tools are constantly a thorn in the side of users and developers alike.
Problems for users
Let's take a typical autoconf package. I'll call it package-xyz. Our user, Joe GNU/Linux User, has just downloaded and untar'd this package. Like most users, the first thing he does is change directory to the newly unpacked source tree and runs a quick 'ls' to see what files are there. To his delight, he discovers a "configure" script, indicating that he probably doesn't have to do any editing of Makefiles or other such craziness. Little does he realize the troubles awaiting him.
Now, Joe again does the typical thing and runs "./configure --prefix=/opt". The configure script runs for a while, and then drops out with an error which basically translates to: "you have an autoconf version that is three weeks old, please upgrade." But this is displayed in the most cryptic means possible. He won't realize this is indeed what the error message means until he runs a few quick google searchs. Now, Joe really wants to install this program, so he doesn't given up that quickly. A few minutes later, he's run apt-get upgrade (or run whatever auto update his distribution uses).
Now, Joe decides that he would like to customize his package. So, he runs "configure --help". A list of options puke themselves out all over his screen. Undeterred, he runs configure again, but this time pipes the output to his favorite pager.
Now Joe is forced to parse this output, because there is no real standard for the options to pass configure. He wants to use gtk, but how does he do this? Is it '--with-gtk', or '--enable-gtk', or perhaps this time it's '--enable-gnome', or maybe '--with-toolkit=gtk'. In any case, he finally "gets" it, and runs the configure script, customizing the package to his hearts delight. But, Joe discovers he's missing lib-lzw-3.2.3.4. Our user takes the time to go through the same rigamarole with that package, and after a bit of tinkering gets it to install. Of course, after installing the package Joe discovers that configure STILL thinks that the library is not there. He does a bit of investigation and discovers that he needs to delete the file config.cache. That is, if he is lucky and the "cached" option isn't in some other random directory or file. (Joe, at that time would remove the source tree, and start back again from scratch).
The configure script now runs properly and outputs a nice Makefile. So the next thing Joe does is run 'make', and to his surprise configure, for no apparent reason, decides to run AGAIN. But that's OK; Joe sits through the same tests as they run over and over(this time hopefully cached). While looking through the log messages, Joe happens to wonder if there is perhaps, maybe, some way to not have to run through the same 50 test, over, and over, and over. After a while make runs, starts compiling the program, and errors out. Joe now reruns make, this time to see what the error is, while redirecting stderr so he can scroll up and see the original error message out of the 95 presented. To his amazement, while configure picked up that he did indeed have lib-lzw-3.2.3.4 installed, it failed to realize that the header files where located in /usr/include/lzw, and not in /usr/include. At this point Joe has a real problem.
Our friend Joe, is a seasoned *nix user. So, his gut reaction is to start emacs(one thing the FSF got right) and edit the makefile only to discover a 50,000 line long monstrosity. He greps through the file, looking for the right variable to edit. Of course, there are 500 different $(INCLUDE) settings spattered throughout the file. Little does he realize that the variable he's looking for is three directories down and called $(YOUD_NEVER_GUESS_THIS_VARIABLE_NAME_HAHA). But, Joe finally finds it and edits it to the proper value. Only, having done this, configure again decides to run for no apparent reason(even though 'make' skipped it the last five times) and overwrites all this hard work Joe has just done.
Joe is presented with an interesting problem. He realize that he needs to edit something besides the Makefile. But what does he look at, configure.in? Makefile.am? Makefile.in? Makefile.yoyo? Makefile.banana?
By now, the average user has done one of the following: A. given up and tried a different package, or B. shot themselves in the head with a twelve gauge shotgun. Joe is feeling masochistic though, and so continues trying. I'll save you the pain of talking about his later problems, or the pain of hearing about Sally FreeBSD user that wants to configure a package to run under GNU/Linux on her iPaq.
Problems for Developers
Your average developer has no clue what m4 is or how it works. Most Unix people, except the old die hards, have not even heard of m4. That didn't stop the autoconf guys from using it though. (Note: to be fair, m4 is a nice tool if you know how to use it.)
But here's the issue, in order to be able to write your autoconf setup files properly(without a considerable amount of pain and suffering), you must first learn m4. We will follow a GNU/Linux developer, a creative lady named Jane. Jane is a seasoned developer. She is accustomed to writing Makefiles, but decides she wants that extra degree of portability that autoconf allows. She isn't familiar with m4, but thinks she can wade through building her files anyway.
Her first action is to look for a tutorial on how to use autoconf. Sadly, she will spend a lot of time looking. If she is lucky, she'll discover a 5 year old online copy of a book that might help her accomplish 5% of what she needs to do. The GNU info system isn't much help either. The documentation she needs is spread out over 50 different info pages, divided further into 3 different packages. These divisions don't help much, as configure.in sometimes makes the Makefile.am do strange things. Eventually, Jane gives up writing the files and does what every single other developer does with the auto* tools: copies someone else. There are perhaps 30 developers world wide that understand the autoconf tool chain. (I doubt there is anyone that quiet understands the brain-dead Makefile that is puked out.) Indeed, this very problem is what caused much of Joe's issues. Well written, configure scripts are rare things of beauty. (They are mostly found in GNU projects, but I've seen a few others.)
The Root of the Problem
The autoconf/configure system is clever, but in the end it is just a very creative hack to work around the deficiencies of Makefiles. Those that have used *nix for a long while have seen several of such work arounds. There is the Sleepy-Cat libdb approach of having 50 different Makefiles, one for each different architecture. There's also the "config.h" approach seen in several different softwares. Then, there is "imake" used by X11 projects. (imake is perhaps the only thing that makes autoconf look sane, but I digress.)
Here, we see why autoconf resorts to outputting 50,000 line long Makefiles. Makefiles lack support for complex dependencies, or complicated error checking. Now, many of the above tools use the same hacks. Things like using "touch" to create files used for dependencies, with complicated build rules on making that file. In any case, a paper written on the various Makefile techniques could feel volumes.
Solutions
There are several unique and innovative things going on currently to help aid the quagmire in build processes.
Two such solutions are "scons" and "cons". Both take a look at replacing Make with something far more flexible. However, they both depend on tools that are a bit less standard that sh and m4. (Those being Perl and Python.) Yet, both languages are quickly becoming standard on the major Unixes. (Find more information about scons or cons: http://www.dsmit.com/cons/ or http://www.scons.org). Scons is a bit less mature, but is prefered by the author of this paper. Using either piece of software allows a much more clear build process for complicated software. There is no requirement of generating a set of build instructions from a template.
Another interesting piece of software is AAP(http://www.a-a-p.org). Here, the user can create "recipes" for building their software.
Any one of these three solutions is cleaner and easier than autoconf. I like the familiarity that Make brings to the table, but it's time to face up to the fact that Makefiles introduce far more complexity than need be.