The screen on a Tablet PC is a standard LCD screen with an electromagnetic (rather than a resistive or an electrostatic) digitizer overlay. This overlay creates an electromagnetic field, so when the pen comes in contact with it the hardware is able to determine exactly where the pen is and the amount of pressure the user is applying. The hardware then passes this information off to the operating system as events, which the OS renders as ink on the screen.

The digitizer samples the screen at 130 samples per second, which enables Microsoft Windows to display and store the ink at very high resolution. This resolution not only insures that the ink appears to flow in a very natural form, increasing the legibility, but it also maximizes the accuracy of the handwriting recognition.

Although the Tablet PC's handwriting recognition has received a great deal of public attention, it's important to remember that there is often no need to convert ink to text to save it or share it with Windows applications. In this case, users simply enter ink into an application, where it's treated as a native data type across Windows-based applications; handwritten data has the same value and versatility as data that has been keyed in on a keyboard. For example, say acoworker has forwarded me a chain-mail or some spam (or worse yet, a virus), and I'd like to add a personal touch to my response. Rather than trying to limit my flame to nothing but ASCII, I can bring the full power of my handwriting to bear while I craft my response. Because ink is treated as a data type, even users who don't have Tablet PCs will see my handwritten response. In other words, I can use Reply All to maximize the humiliation of the original sender.

My lighthearted example notwithstanding, the Tablet PC is certainly going to change the way we approach everything from note taking to data entry to content creation. While hardware advances have received much of the initial focus, it's great software that makes these cool new gadgets useful. Now that the Tablet PC is shipping, it's time for us to start thinking about how we can leverage this new technology while we design and build applications for our users.

Fortunately for us developers, Microsoft put a lot of thought into the new operating system, so things just work for developers. First off, because Microsoft Windows XP Tablet PC Edition is a superset of the Microsoft Windows XP operating system, all our existing Windows applications that work today on Windows XP will continue to work on the Tablet PC-just better. For example, the standard Windows common controls (Comctl32.dll) on a Tablet PC now understand the concept of left- and right-handedness, so ToolTips and menu items pop up on the opposite side of the pen. As you can see in Figures 1 and 2, after a user has configured the machine for right-handed input using the Tablet and Pen Settings in Control Panel, these items appear on the left side rather than on the right side as they do with the standard version of Windows. As long as your applications are using these standard controls, your applications will inherit this new functionality for free.

Figure 1. ToolTip for right-handed users.

Figure 2. Menu for right-handed users.

While somewhat trivial, this example demonstrates the need to review the way we look at user input. Back in 1986 when the first version of Windows shipped, the initial applications were nothing more than crude ports of their DOS forerunners. Soon newer (and ultimately more successful) applications began adding support for drag-and-drop, right-click, and mouse-overs-all things that make perfect sense in a GUI environment but that have no corollaries in a command line world.

When I built my first few ink-enabled sample applications, I found that the types of pen-based user input fell into three categories:

• Text recognition-This is the most basic form, in which the Tablet PC provides nothing other than an alternate means of entering text. Here the main benefit is greater usability for forms-based applications and many types of workflow applications and easing the burden of data input for users of non-Latin-based languages, for which entering a single character often requires many keystrokes. It does not, however, change the way users interact with their computers other than swapping the keyboard and mouse for a pen.
• Free-form handwriting-Here the Tablet PC acts as a never-ending stack of digital note paper. Because these note papers are now digital documents, they can be stored and sorted, thereby providing easy access to information that was once lost on a combination of sticky notes, the backs of meeting agendas, spiral notebooks, and the proverbial restaurant paper napkin. This ability to categorize and retrieve information when combined with the background text recognition that enables you to search for specific words in your handwritten entries does provide a new model for application design.
• Handwriting attached to other document elements-For me, this is the most exciting opportunity for building a new class of applications. I suspect it will also prove the most difficult. We already have some similar functionality in the comment feature of applications such as Microsoft Office; however, the ability to add free-form handwriting provides a much richer user experience. Let's look at this case in more detail and examine the opportunities and challenges of annotating documents.

In this case, users create or revise standard documents as they always would, but now they can add handwriting just as they can with printed versions. Even though I've been composing documents directly at the keyboard ever since word processors were dedicated machines rather than a type of software, when I'm revising my first draft or collaborating with a colleague I'll often print out a copy and use a pen to note desired changes and comments because I can be much more expressive with my handwriting than with ASCII.

For my own editing, the downside is obvious-I need to key in my changes, and it's pretty inefficient going back and forth between the screen and the paper. In collaborative situations with more than one reviewer, the process completely breaks down because the poor person collecting everyone's input now needs to consolidate many different (and usually conflicting) edits on many different pieces of paper.

With digital ink, many of these problems go away because now the annotations are stored with the original document. The benefits are obvious-no more neck strain going back and forth between paper and screen; version control is possible; routing and distribution are easier; and finally the process of creating and annotating is unified. Better yet, the Tablet PC opens up opportunities to annotate new document formats that don't lend themselves to traditional printing, such as audio and video. Imagine bringing NFL-style whiteboard commentary to your training video.

There are, however, some challenges that make developing these applications exciting. The ability for users to annotate their documents raises their expectations, and our new software will need to consider scenarios that we didn't have to worry about before, such as anchoring these annotations and discerning their context.

When users add comments to documents, they're usually commenting on a specific element-a word or sentence, a figure, or even a formatting style-and not the entire document. Therefore, to make digital ink usable, you need to anchor the annotation to the element so that it flows with it while the user continues to edit the document. Actually, the process of anchoring is not that hard; all you need to do is add some metadata to the element that points to the ink data structure that forms the comment. The real challenge, as I see it, is devising a non-intrusive user interface that enables users to identify which element they wish to comment on. In many applications, this might be as easy as highlighting or circling the element; for other applications, however-especially those that don't have discrete elements, such as graphics packages-you must use a different approach.

The really exciting ink-enabled applications take into account the context of the annotation. For example, when a user places a check mark next to an item in a bulleted list, this could signal the completion of a task. An application that understands this context could insert the completion date, change the font color to grey, and move the item to the bottom of the list. In the customer relationship management (CMR) example, highlighting a customer entry might cause the application to replicate that account down to the user's laptop so that it would be available off line. The application I'm looking forward to is one that understands standard proofreading symbols and automatically applies the edits.

Don't let these challenges overwhelm you. They're no different from the standard design challenges you face when building an application and deciding on the proper user interface. In fact, building Microsoft .NET applications for the Tablet PC is surprisingly easy (you can also build COM and Microsoft Win32� apps if you want). In fact, you don't even need a Tablet PC; all you need is the Tablet PC Platform SDK. The SDK includes all the extensions and utilities you need to develop and debug Tablet PC applications on any computer with Windows 2000 or Windows XP Professional. You can use a mouse or a digitizer pad for handwriting input.

Only Tablet PC systems contain the recognizers required to convert handwriting into text, so while you can capture ink with the InkCollector and InkOverlay objects, you won't be able to perform handwriting recognition on it.  Just because you can't perform the actual recognition on a standard PC, however, doesn't mean you can't develop text recognition applications, as I'll show you next.

 

After you've downloaded the Tablet PC Platform SDK, the next step is to create your first ink-enabled sample application. In Figure 3, you can see the sample WinForms application I created.

Figure 3. Sample text recognition application.

Before you start using the Tablet PC API, you need to add a reference to the Microsoft Ink components in Microsoft.Ink.dll (located in %ProgramFiles%\Microsoft Tablet PC Platform SDK\Include\ by default) and then reference the namespace:

// TabletPC API namespace
using Microsoft.Ink;

In my application, I used the InkCollector object, which captures ink from any Tablet PC input device. If you have a Tablet PC, you can write directly on the screen, or you can use the mouse (or a digitizer pad) on a standard PC. To start off using the InkCollector  object, you first declare a module-level variable:

// Variable to collect ink
private InkCollector....m_ink ;

Next, decide which control you want to act as the writing surface. I used a Group Box, but you can use anything that is able to receive WM_SETCURSOR messages. (If the control is on a dialog, you'll need to set the SS_NOTIFY style because the parent filters the messages by default.) When the form starts up, you'll need to attach the m ink variable to your surface. You do this by passing a window handle into the constructor:

// Create a new ink collector and hook it
// up to the group box (you can use most
// any control)
m_ink = new InkCollector(gInk.Handle);

Finally, you need to enable collection:

// You must enable the ink collector
// before it will respond to the pen/mouse
m_ink.Enabled = true;

You now have an application that collects ink. The next step is to recognize the ink and convert it to text. Remember, this works only on a Tablet PC, but I'll show you a technique to handle cases in which you're running your application on a standard PC.

In my application, I created a specific button that enables reconciliation for simplicity, but you might want to have your application use a background thread to automatically convert ink after the user has paused input or left a gap that you interpret to be a space.

The first thing I do in the click event of my button is retrieve a list of recognizers installed on the system. I then test the result to see whether any are installed. If I don't find any, I display a message box warning the user; otherwise, I perform the recognition. This is where the real beauty of the Tablet PC API comes in: to convert ink to text, I simply call ToString() as shown in the following example:

// Get collection of handwriting recognizers
// installed on the machine.
Recognizers recog = new Recognizers();< /FONT >

In this article, I started off with a discussion of how the Tablet PC works and how some of the hardware advances make it easier to develop great applications for this new form factor. Next, I showed you that the Microsoft Windows XP Tablet PC Edition operating system has many enhancements that insure that your existing applications "just work" on the Tablet PC without any modifications. I then discussed three ways that developers can add support for ink in their applications (text recognition, free-form entry, and annotations) and talked about the way that applications change when developers start building ink-enabled applications. Finally, I walked you through the source code of an application that performs text recognition.

As I mentioned at the beginning of this article, the best place to learn about developing for the Tablet PC is with Building Tablet PC Applications by Rob Jarrett and Philip Su. Their new book provides essential details on the TabletPC Ink SDK, including tips and tricks on how developers can produce the most powerful and natural pen-based applications. Because the book is cowritten by the lead software developer on the Microsoft TabletPC team, you can be sure that this is the definitive book on the Tablet PC.

Microsoft Press� provides in-depth documentation for these and all the other issues related to developing for .NET. For a complete list of .NET titles from Microsoft Press, see the Inside Information About Microsoft .NET page. For a broader overview of all the Microsoft .NET technologies, you should look at David S. Platt's Introducing Microsoft .NET. It has some great information on COM Interop, Windows Forms, and .NET memory management.

You'll also want to investigate these MSDN� resources: