NXTender: Extend Your NXT

By Sivan Toledo
September 2006

The tray-icon and popup menu of NXTender How about adding a really powerful extension to your NXT? I'm refering to your PC.

What is NXTender?

NXTender is a program that your run on your PC and which lets your NXT control your PC. This program allows the NXT to send keyboard and mouse events to the PC (simulating a keyboard or a mouse), to request that events from a joystick connected to the PC be sent to the NXT, to open windows on the PC and to display data in them, and so on (the so-on part is currently just a wish).

The Philosophy of NXTender

To be productive and to have fun, NXT builders should be programming the NXT, not the PC.

In my first two NXT-PC projects (the NXT as a remote two-key keyboard and the joystick-controlled ultrasonic explorer), I designed a custom communication protocol for each project and wrote a custon PC-side program for each. I quickly realized that this is pretty hard. For example, if your program sends both text messages to the NXT and numbers (say joystick positions), then it's best to send them to different mailboxes on the NXT. So your NXT program and your PC program must be coordinated in terms of the mailboxes that are used for each type of data. As you add or remove things that are sent, you must keep changing both programs.

I decided to create a PC-side program that will allow you to contoll the NXT-PC interaction from the NXT side alone. This would allow NXT builders who are not proficient PC programmers to use the PC as an extension of the NXT. It's a powerful extension, given the computational power of PC and the range of things that connect to them (joysticks, keyboards, mice, printers, scanners, cameras, cup warmers, and anything connected to the internet).

Let's take joystick input as an example. With NXTender, if your NXT wants joystick input, it tells the PC to send joystick events to the NXT, and it tells the PC which events it wants to listen to (say, only Y-axis and button events) and in which mailboxes it wants to receive them. This means that you can control all the relevant details of the NXT-PC interaction from your NXT program.

Let's take radar-style displays as another example. With NXTender, the NXT can tell the PC to open a radar window on the desktop, it can tell the PC to display dots at certain angles and distances from the center of the window, it can tell the PC to clear the dots from the window (for the next scan), and it can tell the PC to close the window. Thee window is not always there; it's there only when the NXT wants it there. 

Plugins

Suppose you want to build a NXT robot that will control a USB cup warmer or lamp connected to your PC. Can you do this with NXTender? No, this require some custom programming that will control these devices. The standard version of NXTender does not know how to controll them. But I have tried to make it as easy as possible to add this custom code to NXTender.

NXTender controls the PC using a set of modules called plugins. Each plugin is responsible for responding to NEXT messages that start with a certain word. For example, in the standard version of NXTender the keyboard plugin responds when the NXT sends a command like KBD LEFT, but the radar plugin responds when the NXT sends the command RADAR OPEN.

What a plugin does in response to a message is entirely up to the plugin. It can do whatever it wants on the PC, and it can send messages back to the NXT. 

So to controll your cup warmer from the NXT, you will have to write a NXTender plugin that responds to commands starting with, say, CUP-WARMER, and that actually controls your cup warmer. How easy or hard it is depends on your programming skills and on how difficult it is to control a particular device from a PC program, but it's easier than writing a complete program that interacts with the NXT, because NXTender takes care of the details of the NXT-PC communications. NXTender also allows you to easily build NXT creations that interact with multiple devices on the PC. For example, if your NXT creation needs both joystick input and to contol the cup warmer, then you only need to implement the cup-warmer plugin; the joystick plugin is built in.

Installing and Running NXTender

Requirements: Windows with java 1.5 or later (also called java 5.0) installed. Sorry folks, it currently only works on windows. You can check your version of java by giving the command java -version on the Windows command line.

Download the zip file and unpack its contents into some directory, say C:\Program Files\NXtender. Now either 

Now you need to pair your NXT with the PC via bluetooth. I always do the pairing from the XNT.

Finally, you need to tell NXTender on which serial port the NXT is connected. Right-click on the Bluethooth icon on the tray, select Advanced Configuration, then the Local Services tab, and note the name of the serial port (it's COM14 on my PC). Now right-click on the NXTender icon in the tray, and select the correct port number from the list. It should appaer in the list. You only have to do this once; NXTender remembers this port number for ever.

To stop NXTender, right-click on the icon try and select Exit.

Troubleshooting: if you try to connect to the correct port and you get an error message, it might be caused by a running NXTender that is still connected to the serial port but has already closed the tray icon. (If this happens, then it's a bug in NXTender.) Open the task manager and kill the running NXTender process, which will be called java.exe.  

Using NXTender (NXT Programming)

There are only three rules that you should know about using NXTender when you create your NXT program:

  1. Send commands to NXTender via mailbox 1.
  2. NXTender only sends messages to the next if the NXT gave a command to one of the plugins to send some data back; therefore, in general the NXT program should not listen to NXTender messages all the time.
  3. The interaction with plugins is plugin-specific; therefore, to use a plugin you need to read the documentation of the plugin, to learn which commands it responds to and how.

Here are the links to the documentation of the built-in plugins and the user-contributed plugins.

How to Contribute to NXTender

NXTender is an open-source project. It is freely distributed with its source code (everything is in the zip file), and you are welcome and encouraged to contribute to it. There are two main ways to contribute to NXTender:

  1. Writing a plugin. Share your cup-warmer plugin or your face-recognition video-input plugin will allow others to benefit from your efforts. (I still need to write a short plugin-writing guide).
  2. Porting NXTender to another platform (Linux, MacOS, etc). Write to me if you plan to do this, since it may require some changes in the architecture of NXTender, and it would be better to keep the code unified. Most of NXTender is written in Java and does not require any porting, but there are also native libraries for communications with the NXT and for various plugins.

A Challenge for Creative Minds

How creative are you? I'm announcing a NXTender challenge: to design and build the best Lego mouse-replacement device (it could be a Lego mouse, trackball, trackpoint, trackpad, whatever, as long as it can function well as a mouse). I've done the PC-side programming for you, now you build the Lego thing.

Appendix (for Java Experts Only): How to Implement a Plugin

  1. Implement a singleton Java class that extends sivantoledo.nxt.NXTender.Plugin:
    public static abstract class Plugin {
      // irrelevant private fields
      // The name argumet is the command prefix, such as KBD
      public Plugin(String name) {
        plugins.put(name,this);
      }

      /*
       * You must implement
       * public static Plugin getInstance(sivantoledo.nxt.Sender sender)
       * otherwise your plugin will not load.
       */
      public abstract void process(String command);
      public abstract void close(); // shut down all threads etc.
    }
  2. The Sender that is passed to getInstance allows your plugin to send messages back to the NXT. Its interface is:
    public abstract class Sender {
      public abstract void send(byte mailbox, String  s, boolean important);
      public abstract void send(byte mailbox, int     i, boolean important);
      public abstract void send(byte mailbox, boolean b, boolean important);

      /*
       * The following methods all send an important message.
       */
     
      public void send(byte mailbox, String  s) { send(mailbox, s, true); }
      public void send(byte mailbox, int     i) { send(mailbox, i, true); }
      public void send(byte mailbox, boolean b) { send(mailbox, b, true); }
     
      public abstract void flushQueue(byte mailbox);
      public abstract void flushAllQueues();
    }
  3. Important messages will not be dropped. Unimportant messages will be dropped if the message queue contains more than 5 messages.
  4. Package your plugin (which consists of the singleton plugin class and possibly other classes) in a jar file.
  5. The name of the jar file must be the fully-qualified name of your plugin class, e.g., org.superobots.nxtender.cupwarmer.CupWarmer.jar.
  6. Put this jar file in the plugins directory of NXTender.
  7. Please consider sharing your plugin with others. This includes (a) that you write it in a general-enough way to be useful to others, who perhaps have slightly different needs or resources than you, and (2) making it available for download.

Appendix: Implementation Details

Change Log

© 2006, Sivan Toledo