The hard and the really hard

kw: computers, software, programming, artificial intelligence

I noted earlier the report that a computer system now exists which exceeds the processing power and memory capacity of a human brain. It just needs about nine million watts of electricity to run. However, if things proceed into the future as they have in the past, in thirty years such capacity will be available in larger desktop personal computer systems, and in a further thirty years, in a pocket device, a successor to the smart phone.

There are good reasons to think that future progress may not follow the trend of the past half century or so. Moore's law may be running out of steam. There are several versions of the "law", actually a well-defined trend. The original trend identified by Gordon Moore in 1970 states that the number of devices on a CPU chip tends to double about every two years. In 1971 the 4004 CPU had 2,300 transistors on-chip. In 2011 a 10-core SPARC processor had about 2.6 billion. That is a factor of 1.13 million in 40 years, or just over 20 doublings. So that element of the law has been working just fine. I wonder, though, whether just another ten doublings (a factor of about 1,024) can be accommodated: 2.7 trillion transistors on one chip? On a watch-sized chip (4 sq cm), that is 150 square nanometers per transistor, or a feature size in the 10-12 nm range. That's where it gets hard to keep electrons going where they are supposed to, because of Heisenberg uncertainty.

Other elements? Performance does show signs of hitting a limit. Let's look at a fifteen-year span that is well studied. In 1994 the first Intel Pentium chip was introduced. At 75 MHz, its benchmark speed was 12.2 MFlops. Seven years later, the Pentium 4 ran at 1.7 GHz and benched 152 MFlops, 12.5x faster. From 2001-2009, CPU clock rate didn't quite double, to 3.07 GHz in turbo burst mode in a Core i7, but the benchmark (per core) increased to 667 MFlops, an increase of 4.34x, mainly due to better architecture. The benchmark doubling time in the first seven years was 1.9 years, while in the latter eight years, it was 3.8 years. In the 2006-2009 time frame, doubling time was more like seven years. But now the norm is four, six or eight cores on a large die, making single-thread codes less relevant. I don't expect single-core benchmark speeds to much exceed 1,000 MFlops for some years to come.

To me, all this means that getting the power of the brain into a watch-sized hunk of silicon or a successor material is going to take longer than we might predict, based on the past fifty years of computer hardware history.

There is a second hurdle in the way of getting useful work out of all that power: software development. Do we want a silicon brain to run the same way our lipid-based brain does? It seems a silly idea to me, but not to many proponents of artificial intelligence. Some people are saying that the Watson supercomputer, by winning two days of Jeopardy!, has passed the Turing test. Not really; nobody was trying to make it fool us into thinking it was human. It won a specific kind of trivia contest, as a machine, using machine methods rather than human ones. It was a successor to the Deep Blue chess match against Gary Kasparov. The computer didn't try to behave as a human would, nor was it in any way disguised. Neither system could navigate its way out of a crowded living room (were it mobile).

I don't have a definite figure, but IBM seems to have spent half a billion dollars developing the software code that makes Watson's hardware a Jeopardy! wizard. It will cost dozens of millions more to re-purpose the Watson hardware into a medical diagnostic machine, because of course, diagnostic medicine is not a trivia game, though it does require the marshaling of numerous loosely related facts.

Watson's software is on a par with an operating system. Even your telephone has an operating system. The popular Android OS for smart phones, according to a recent article, has 12 million lines of program code (plus 5 million lines of comments), in forty programming languages and scripts. Roughly speaking a "language" is converted into machine code before use, while a "script" is interpreted from a more human-readable version each time it is used. The compiler for a language is comparatively small: 150,000 lines of code in the case of the Perl compiler. The real heavyweights are full-scale OS's for computers: Linux has 200 million lines of code and Windows 7 is in the 100 million range (Vista had 50 million).

Here is where the kind of CPU you are using has some influence. Much of the code of an OS is in assembly code, and a "line" of assembly code does more on an Intel CPU than on one designed to run UNIX or Linux. So that 100 versus 200 million difference is smaller than it looks.

What does a line of code cost? It depends on the kind of code, but IBM long ago found that a "good" journeyman programmer could write and debug three lines of code daily. A small number of superprogrammers (I was one for thirty years) can do ten to 100 times as much code writing. In FORTRAN, I typically produced 50-100 lines per day. In assembly code, I produced half as much. During the last years I was an active programmer, I earned around $20 per hour, but my work cost my company $50 per hour with overhead, or $400 per day, so a line of my code cost in the $8 range. The larger teams of programmers needed for huge projects like Windows 7 typically include very few superprogrammers, so even with more efficient methods of code generation that are possible using "Visual" languages, a line of code costs $50-100.

Put the figures together. It cost close to a billion dollars to develop Android, and ten times that much to develop Windows 7. That's why Microsoft has to charge $100 to $400 for a copy of the OS, and hope to sell 100 million of them. The first 70-80 million copies just pay the development costs.

Now, consider the human brain. To duplicate all its functions might take billions to trillions of lines of code, if we go the software development route. 'Taint gonna be cheap! Of course, as with an OS, you only have to do it once. But the romantic notion that a lone programmer somewhere will develop a "soul in silicon" is just not in the cards. One of my colleagues was ten times as productive as I was: 500-1,000 lines of good FORTRAN daily (that's a lot of typing, each and every day). So a million lines of code would take him 1,000-2,000 work days. That's four to eight work years. Ten such programmers could produce Android in ten years or less. The actual Android crew, numbering much more than ten, took two years. I tip my hat to them.

Now that we're on the verge of software projects that might be of human-brain scale, can it be done? First, you have to know what you actually want. What would success look like? Right now, if we wanted to start programming "consciousness", we'd be in a position like these folks:

You lot start coding…

…I'll go find out what they want.

Source: www.abberley.co.uk/asap/

There are ten thousand or more studies of what consciousness is. They can't even agree on two or three basic rules to help them recognize consciousness when it appears. Philosophers have been arguing this for centuries (30-40 of them), without producing anything a computer programming team can use as a target. It is going to be an emergent property of some collection of parallel processes, not parallel as doing the same thing, but each set doing something different. But there is no agreement on what are the necessary processes and which ones are simply tools used by a conscious being.

There is not even agreement about whether a physico-chemical body is required. Our brain's operation is strongly affected by hormone levels, and it may be that "our" kind of consciousness (I am including all mammals and birds here) is intimately related to the body's responses to environment, via its chemical cues. I suspect our real "brain" is not just the 1.4 kg of gray+white matter inside our skulls, but includes the other 40+ kg of the body, or at least the 5-10 kg that comprise our nervous lashup plus our endocrine system. I suppose from the total brain's point of view, most of the body is a support system. But the endocrine system may turn out to be essential for any sort of consciousness that we can understand well enough to converse with.

Oh, there is so much to learn, and lots of eager folks trying hard to learn it. It is fun to watch, even though I am pretty much on the sidelines these days.

Half a million and counting

kw: computers, software

Just a quick note on something that has been taking up my time. I've been wanting to analyze asteroid distributions for some time, and I recently began using Office 2010. Excel 2010 is not limited to 65,535 rows the way Excel 2003 is; its default limit is 1,048,575, and there is a way to make that larger.

Last evening I downloaded the Asteroid Orbital Elements database from Lowell Observatory. It took about an hour, being a 143-Megabyte text file. It includes about thirty parameters for more than half a million minor planets. Just opening it in Excel 2010 and getting the columns set up correctly took a while.

I extracted the columns I care about to another workbook, which then had some 34 Mby. I found that my computer is pretty fast, doing trigonometric calculations on the entire set in about three seconds. I don't know what that is in MFlops, but I was impressed.

I have more selecting to do. I'm separating out two data sets, the major part of the Main Belt and the trans-Neptune objects. Then I'll be able to do the statistical work I've been waiting on. The more I use Office 2010 the better I like it.

Tax season cometh

kw: taxes, software

Every season is getting earlier every year. Stores put up their Christmas displays on November 17. Now here it is December 14, and the tax software promotional from H&R Block arrived. We have been using their Tax Cut software a few years. This year it is called H&R Block At Home™.

This is no software review. You'll get a great review of the top ten tax preparation programs at Top10Reviews.com. This is a product endorsement. To see this sample screen shot, click on the image. The "big" image is still undersized, but gives you an idea of how the screens look.

We used several products over the past nine or ten years, and have settled on this one. The offer we got, at $14.95 including a free e-file, really puts pressure on tax preparation services.

We have a friend who runs a tax preparation company. Their bread-and-butter used to be the yearly January-to-April tax season. Now it is the quarterly Estimated Tax and Business tax filings. Personal taxes have dropped way off. At $125, more and more people are climbing the learning curve so they can prepare and file their taxes on their own computer, often for free. Some who can't afford a computer, but must still file tax forms, use free computers at the library and a free online service (HR Block has one of those also).

We won't get W-2 forms for another month or so, but the tax software folks already got their foot in the door.

Impossible ancestors

kw: genealogy, software

As one might guess from several recent posts, I've been spending a lot of time on my family tree at ancestry.com recently. I finally purchased the companion software, Family Tree Maker, the 2011 version. It has a few specialized tools I've been wanting. One is the Errors Report, which audits a list of conditions, particularly date anomalies.

When I first began loading the family data, the web site had an option to take "hints" that included copying up to five generations from another public tree or from the One World Tree, which is a semi-curated meta-tree. Once I had entered the people and relationships I knew already, I happily took hints pretty freely. I got a lot of junk in the process. I am glad that the download options are much more restricted now, a generation at a time, with the chance to check each person before clicking OK.

Anyway, the Errors Report produced a five-page list of people, mostly those who are recorded as having married or had children prior to their 13th birthday, after their death, or past childbearing age. I'll go through the targeted sections of the tree and remove those that are most suspect, unless there is new information with better dates. While I like to collect ancestors, I don't want to collect imaginary ones!

Flash! - See new Google Maps

kw: news, software

OK, you can't quite see the words "Georgia Aquarium" in this image, but you can in the full size clip you'll see by clicking. The "Earth" enhancement to Google Maps just came out today, in the past hour or two. It is Google Earth running inside Maps. You can get directions, then have the blue track shown in 3D, with 3D buildings in the cities. The green symbol is the end of a trip I plan to take in a few weeks, to see the new aquarium in Atlanta.


Google Maps has become indispensable to me in recent years. This adds the ability to preview in even more realistic ways. I can take a look at a hotel I might stay at, if it isn't already shown on Street View. As in this case, I can verify that the target shown is in the right place. Very cool!

Getting around a Picasa limitation

kw: observations, experiences, software

Those of us who use Picasa from Google to manage our photos got a recent upgrade, which some folks have been none to pleased about. About ten days ago I also had an anomaly with it, but I found a way around it.

If Picasa is already managing thousands of photos, and you upgrade to version 3.5 or 3.6, which has the new face recognition feature, it does not complete its face indexing task. In my case, it trundled along for about twelve hours, got to the point where it was 54% done, by its own estimation, stating that it had indexed more than 9700 faces and had 8800 or so to go, and it stalled. All the counters stopped moving, hard disk activity slowed and stopped. I had been putting names to faces for a few hours as it worked, and I kept doing so. But it didn't work through any of the remaining faces.

I let it sit there for the next twelve hours in that condition (I just let it run all night). No change. At that point, I closed the program, shut down the computer, and went to work. When I started Picasa up that evening, it was waiting for me to continue to identify faces. I spent several hours finishing the ones it had found, and that was that. I could tell it had missed quite a few.

A few days later, the big snowstorm came (Feb 6), and I was snowed in for two days. I took the chance to copy my whole photo archive to a newer computer. There were just a few photos on this one, and Picasa 3.6 had already scanned for faces, and I'd identified them. I got the DVD's out with my photo collection, and loaded a year at a time. I would load a year, then Picasa would gather the photos, scanning for faces. The first year, it left 55 faces untouched, even after I had put names to all the others. I just loaded the next year, to see what would happen. This time, it cross-checked them all, and thereafter, it always completed its face checking. I did the name tagging year-by-year.

So when I was done, I had every face either name-tagged or Ignored. The process took all of one day and a little of the next. This is the work-around then: move your photos to data DVD's first, then upgrade Picasa. Put them back a thousand pix or so at a time. Organizing by years makes that a simple process. You'll need a couple hours to tag each year's photos (if you are as active a photographer as I am).

I must say, other than this limitation, I like how this feature works. Picasa's face-matching routines let it present you with a face that often represents a group of similar views of the same person. By tagging the one you see, you tag the whole group. The more faces you tag, the more accurate its picks become. Of course, you still need to go through each Face Album later to find mis-tagged faces and re-tag them. This is a one-thumb-up upgrade.

It can take a lot more pixels than you think

kw: photography, techniques, software

Photography, like realistic painting, is the art of getting onto paper some facsimile of what the eye sees. Until you try to get "all" aspects of a scene to match what you could see, it is hard to realize just how powerful human vision can be. While our vision system is only in sharp focus over a rather small area—a few degrees wide—we rapidly scan that focus over a scene and our memory stitches it together into a comprehensive view. Reproducing a single still image with that comprehensive view is a challenge. In the discussion below, I'll mention software that increases the areal extent of a digitally imaged scene (panorama stitchers), that increases the apparent dynamic range (HDR programs), and that allows the display of extreme depth of field (focal stitchers).

Firstly, we can swivel our focus over ±70° right and left and ±50° up and down, without moving our head, and we routinely scan an angular field that covers about a tenth of the maximum area, perhaps a thousand or so square degrees. Average resolution in the visual center is about 1/60 degree, so 60x60 is 3,600 pixels per square degree. That doesn't seem like too much: about 3.6 Megapixels (Mpx). A photo that covers a 39°x26° span with 60 pixels per degree just matches what we "usually see", discounting peripheral vision.

A quick swivel of the head, though, allows us to gather a much wider span, and we tend to remember the whole panorama as a unit. The more we gaze, the more we see, and we frequently remember a "sight" that covered 150°x60°, containing 30 Mpx or more.

For generations, photographers have used scissors-and-paste methods or darkroom techniques to put together panoramas that covered all of a scene. These days, software can do this for us. I remember the awe I felt when first I stitched a scene together. I used the "Make Panoramic Photo" option in Windows Live Photo Gallery (there is a link to download it here).

There is also a Photo Stitcher program that comes with any Canon camera, and there is an app for an Iphone, which was used to make the montage shown here. (All the images in this post are shown rather small; click on any of them to see a larger version). This montage shows, at the top, two of at least four images used to make the raw stitch shown at the bottom. One would normally crop out a rectangle to your taste from the rather blobby stitched image.

The Canon software has some hand controls available, and requires hand work if your collection of images don't follow a single "straight line". I've used the Windows Live tool to stitch as many as nine photos in a 3x3 matrix, and as many as ten that were in a line, which it did without further input from me beyond choosing the images I wanted stitched. You just need plenty of overlap, 20% or so in each direction.

You can also get large panoramas from a costly fisheye lens on a high-dollar camera with 20+ Mpx camera such as the Nikon D3X. Stitching is cheaper. So this helps us catch a scene as big as the eye can see.

Then, there's the dynamic range conundrum. Film photography helped us out by recording dynamic ranges of up to 1000:1 in a good negative, and we could use print paper with various amounts of contrast to reduce any or all of this range to the 50:1 range that can be reproduced on a good print. 1000:1 is ten f-stops (actually, 10 stops is 1024:1). But the eye can see, in any glance, a range of more than 15,000:1, perhaps fourteen stops. And when we look for a moment into a brighter or darker part of the scene, the eye quickly adjusts over another five or six stops (~50:1) to allow us to see details nearly everywhere.

In this image, the eye sees at least as much as is shown in the left panel, while a "normally exposed" single digital image is stuck with that on the right, which has a dynamic range of no more than 256:1; and actually, since digital is a linear medium, some of the range is wasted, and we don't see all 256 shades, but more like about 80-100. For many scenes that is OK, but of course we often want more.

Software such as EasyHDR Basic (available free here), takes information from three or more images that were exposed at different levels, to produce one range-reduced image that looks more like what the eye could see. The common technique is to use the Bracketing method that many cameras now have in them, which exposes three times, once "normal", once two stops faster, and once two stops slower. Some cameras even will combine these in to an HDR image automatically.

Adding a total of four stops more brightness information to your scene takes a 256:1 range and expands it to 1024:1, if you are getting JPG files from your camera. If you have the ability to get RAW format files, they start with either 4,096:1 (12 bit) or 16,384:1 (16 bit), so it is better when shooting RAW (which EasyHDR can handle), to go manual and shoot at least four stops over and under for your added images in the stack you'll be using. Of course, be sure to do this using a tripod. Though the software can shift the images to overlap better, the result will lack some sharpness if you start with handheld images. I understand that qtpfsgui has better control of image offsets. Haven't tried it.

So what do we have at this point? Panorama software that can make our pictures taller and wider as needed, and HDR software that can increase the detail we see in the highlights and shadows. There is one more step (so far…).

When your eyes scan a scene, they automatically focus, so your memory of the scene is sharp everywhere, with rare exceptions. A camera's depth of field depends on two things: focal length and aperture (f/ratio). Wide-angle lenses are most forgiving; at a specific f/ratio (say, f/8), a lens of half the focal length will have twice the depth of field at mid-range, and more than that near the hyperfocal point (which is the focal distance to set so that things at "infinity" are just in focus, and a range of closer objects are also seen sharply).

There are trade-offs, however. In small cameras, such as most point-and-shoot, the pixels are so small and the lens is so small, that while there is a large focal depth, the image is slightly unsharp everywhere. When you print a 10 Mpx file on 6x4 paper, it looks good, but enlarging it past 7x5 can show up some fuzziness. In SLRs, with their much larger sensors, the lenses are also larger, and their focal lengths much longer. For example, my Nikon D40's lens has a focal range of 18-55mm; my Canon SD1200's lens has a focal range of 6.3-18.9mm.

So far so good. Because its lens is so short, the Canon's range of f/ratio is f/3.5 to f/8, and no further. The Nikon's is f/3.5 to f/32. One might think, "Great, the Nikon can take really deep-focal-range pix", and it can, but with it there is an overall fuzziness due to diffraction. Roughly speaking, the smallest spot a camera can record (in the sensor plane) is the wavelength of light used times the f/ratio; for f/32, this is 32x0.5µ, or 16µ. The sensor's pixels are 7mm apart, so a 6 Mpx image taken at f/32 has no more resolution than a "perfect" image of 1.5 Mpx, or even less. A ratio of f/11 is about as far as one can go for an image that is sharp at least somewhere!

Helicon Focus to the rescue. This software (available here) takes several images shot at different focus distances and combines the sharpest parts of each into a single image with "infinite", or at least, "extreme", depth of field. The promo photos showing the fly's head and the shotgun illustrate what is possible. It takes three to ten or more images to produce one XFD (I just made this up, for eXtreme Focal Depth) image.

Now, the eye does all this at once. What are we to do?

One reviewer noted that one piece of HDR software (he didn't say which) can also make panoramas. Presumably, you take two or more sets of your panorama images and the program both stitches and applies HDR composition to get an image like this one.

This is cool. It also takes lots and lots of images. This image alone used at least three angle settings to cover the width of the view, and at least three images of each angle setting, for nine starting images. I assume this was made at a setting similar to f/11, so there is decent focal depth.

To get real XFD in such an image, you're getting into a combinatorial explosion! At the very least, 27 images to start with, assuming you had a program that could do all three things at once!! The original image for this HDR panorama is about 2500x800 pixels, or just 2 Mpx. But it took at the very least nine 1.5 Mpx images as a starting point, and I'll bet that the original camera has 6 Mpx or better (54 Mpx or more for the progams to grind through).

To recap: A well-framed and -exposed photo with almost any modern digital camera can record most of what the eye sees over an angular area that one tends to take in "at a glance". To take in all that the eye can see over the area of a quick side-to-side scan requires lots of images. At the very least, 2x2x2, or eight, and a more likely "usual" stack of 3x3x3, or 27. There is plenty of room for improvements in photographic techniques and in the software needed to support them.

One way to sidestep the XFD situation is to use a Gigapixel method (there are several, most using robotic image capture), in which a very large image is stitched from a multitude of small images. Each can be well focused in its small area, so the result will appear to have extreme focal range. Since the Canon software can stitch together as many as 26 images (and I think the Windows Live tool is similarly capable), careful panorama planning can produce large photos that are sharp nearly everywhere. But just think, if you want to go HDR with 26 starting "pointings"; it'll take at least 78. I think the right order of processing is to make the three panoramas, then do HDR processing. The other way around, you'd likely have variations of final lightness. The mind boggles.