#35 Teaching a Toddler
I have a one-and-a-half-year-old daughter, Grace. She has known for some time now that typing on the computer is something that Daddy does for fun.
Whenever I'm writing, she will come over to me, smile sweetly, climb up on my lap, and pound the heck out of the keyboard. (Thank God for xlock and early bedtimes.)
To help her learn how to use a computer, I wrote a simple Perl script that displays a picture and plays a sound whenever a key is pressed. For example, press B and a picture of a bee appears as the word bee is spoken. Press C and a cow appears, D and a dog appears, and so on.
It quickly became apparent that even this simple program was too complex for her. After all, she can't recognize letters just yet. So I modified the program to allow for an even simpler mode of operation. Press any key and you get the first letter of the alphabet (both displayed and spoken), press another and you get the next letter, and so on.
The result is a game that she loves and can play for up to half an hour without stopping. Actually, she can play it longer, but after half an hour my wife and I get sick of hearing the same set of letters and words over and over again and redirect her energy toward the LEGOs.
The Code
1 #!/usr/bin/perl
2 #
3 # Display a big window and let Grace type on it.
4 #
5 # When a key is pressed, display a picture and
6 # play a sound.
7 #
8 # The file cmd.txt contains the sound playing
9 # command.
10 #
11 # The format of this file is:
12 #
13 # key <tab> command
14 #
15 #
16 use strict;
17 use warnings;
18 use POSIX qw(:sys_wait_h);
19
20 use Tk;
21 use Tk::JPEG;
22
23 my %sound_list = (); # Key -> Command mapping
24 my %image_list = (); # List of images to display
25
26 # List of sound commands in sequential mode
27 my @seq_sound_list;
28
29 # List of images in sequential mode
30 my @seq_image_list;
31
32 my $bg_pid = 0; # Pid of the background process
33
34 my $canvas; # Canvas for drawing
35 my $canvas_image; # Image on the canvas
36
37 my $mw; # Main window
38 my $mode = "???"; # The mode (seq, key, debug)
39
40 #
41 # Called when a child dies.
42 # Tell the system that nothing
43 # is running in background
44 #
45 sub child_handler()
46 {
47 my $wait_pid = waitpid(-1, WNOHANG);
48 if ($wait_pid == $bg_pid) {
49 $bg_pid = 0;
50 }
51 }
52
53 # What we have to type to get out of here
54 my @exit = qw(e x i t);
55 my $stage = 0; # How many letters of "exit" typed
56
57 my $image_count = -1; # Current image in seq mode
58 my $sound_count = -1; # Current sound in seq mode
59
60 #################################################
61 # get_image($key) -- Get the image to display
62 #
63 # Make sure it's the right one for the mode
64 #################################################
65 sub get_image($)
66 {
67 my $key = shift; # Key that was just pressed
68
69 if ($mode eq "seq") {
70 ++$image_count;
71 if ($image_count > $#seq_image_list) {
72 $image_count = 0;
73 }
74 return ($seq_image_list[$image_count]);
75 }
76 return ($image_list{$key});
77 }
78
79 ##################################################
80 # get_sound($key) -- Get the next sound to play
81 ##################################################
82 sub get_sound($)
83 {
84 my $key = shift; # Key that was just pressed
85
86 if ($mode eq "seq") {
87 ++$sound_count;
88 if ($sound_count > $#seq_sound_list) {
89 $sound_count = 0;
90 }
91 return ($seq_sound_list[$sound_count]);
92 }
93 return ($image_list{$key});
94 }
95 ##################################################
96 # Handle keypresses
97 ##################################################
98 sub key_handler($) {
99 # Widget generating the event
100 my ($widget) = @_;
101
102 # The event causing the problem
103 my $event = $widget->XEvent;
104
105 # The key causing the event
106 my $key = $event->K();
107
108 if ($exit[$stage] eq $key) {
109 $stage++;
110 }
111 if ($stage > $#exit) {
112 exit (0);
113 }
114 # Lock system until bg sound finishes
115 if ($bg_pid != 0) {
116 return;
117 }
118
119 my $image_name = get_image($key);
120 my $sound = get_sound($key);
121
122 #
123 # Display Image
124 #
125 if (defined($image_name)) {
126 # Define an image
127 my $image =
128 $mw->Photo(-file => $image_name);
129
130 if (defined($canvas_image)) {
131 $canvas->delete($canvas_image);
132 }
133 $canvas_image= $canvas->createImage(0, 0,
134 -anchor => "nw",
135 -image => $image);
136 }
137 else
138 {
139 print NO_KEY "$key -- no image
";
140 }
141 #
142 # Execute command
143 #
144 if (defined($sound)) {
145 if ($bg_pid == 0) {
146 $bg_pid = fork();
147 if ($bg_pid == 0) {
148 exec($sound);
149 }
150 }
151 } else {
152 print NO_KEY "$key -- no sound
";
153 }
154 }
155
156 #################################################
157 # read_list(file)
158 #
159 # Read a list from a file and return the
160 # hash containing the key value pairs.
161 #################################################
162 sub read_list($)
163 {
164 my $file = shift; # File we are reading
165 my %result; # Result of the read
166
167 open (IN_FILE, "<$file") or
168 die("Could not open $file");
169
170 while (<IN_FILE>) {
171 chomp($_);
172 my ($key, $value) = split / /, $_;
173
174 $result{$key} = $value;
175 }
176 close (IN_FILE);
177 return (%result);
178 }
179
180 ##################################################
181 # read_seq_list($file) -- Read a sequential list
182 ##################################################
183 sub read_seq_list($)
184 {
185 my $file = shift; # File to read
186 my @list; # Result
187
188 open IN_FILE, "<$file" or
189 die("Could not open $file");
190 @list = <IN_FILE>;
191 chomp(@list);
192 close(IN_FILE);
193 return (@list);
194 }
195 #=================================================
196 $mode = "key";
197 if ($#ARGV > -1) {
198 if ($ARGV[0] eq "seq") {
199 $mode = "seq";
200 } else {
201 $mode = "debug";
202 }
203 }
204
205 $SIG{CHLD} = &child_handler;
206
207 if ($mode eq "seq") {
208 # The list of commands
209 @seq_sound_list= read_seq_list("seq_key.txt");
210 @seq_image_list =
211 read_seq_list("seq_image.txt");
212 } else {
213 # The list of commands
214 %sound_list = read_list("key.txt");
215 %image_list = read_list("image.txt");
216 }
217
218 # Open the key error file
219 open NO_KEY, ">no_key.txt" or
220 die("Could not open no_key.txt");
221
222
223 $mw = MainWindow->new(-title => "Grace's Program");
224
225 # Big main window
226 my $big = $mw->Toplevel();
227
228 #
229 # Don't display borders
230 # (And don't work if commented in)
231 #
232 #if ($#ARGV == -1) {
233 # $big->overrideredirect(1);
234 #}
235
236 $mw->bind("<KeyPress>" => &key_handler);
237 $big->bind("<KeyPress>" => &key_handler);
238
239 # Width and height of the screen
240 my $width = $mw->screenwidth();
241 my $height = $mw->screenheight();
242
243 if ($mode eq "debug") {
244 $width = 800;
245 $height = 600;
246 }
247
248 $canvas = $big->Canvas(-background => "Yellow",
249 -width => $width,
250 -height => $height
251 )->pack(
252 -expand => 1,
253 -fill => "both"
254 );
255 $mw->iconify();
256
257 if ($mode ne "debug") {
258 $big->bind("<Map>" =>
259 sub {$big->grabGlobal();});
260 }
261
262 MainLoop();
Running the Script
The script has three modes:
- key Press a key on the keyboard and the corresponding picture appears. In this mode, the program grabs the keyboard and mouse, preventing Grace from typing in any other window.
- debug Similar to key mode, only without the grabbing. When the program grabs the keyboard and mouse, it's not possible to run the debugger. (The main program has grabbed the keyboard, which prevents you from typing anything in the debug window.) This mode allows you to run the debugger.
- seq Sequential mode, in which a sequence of pictures (with accompanying sound) appears.
To run the program in key mode, just run the script:
$ grace.pl
Seq and debug modes are specified on the command line, as in this command to run the program in seq mode:
$ grace.pl seq
In key mode, when a key is pressed, a picture is shown and a sound played. The files image.txt and key.txt define which pictures and sounds are associated with each key.
The format of the image.txt file is as follows:
key-name
image-file
key-name
image-file
...
For example, here's a short image.txt for the letters a, b, and c:
a image/apple.jpg b image/beach.jpg c image/cow.jpg
The key.txt file uses a similar format:
key-name
command
key-name
command
...
This tells the program which command to execute when a key is pressed. The way the system is designed, the commands should play a sound. Here's a sample file:
a play sounds/sound1.au b play sounds/seasound.wav c mpg123 sounds/Cow02.mp3
Note:
The system was designed this way because there are a lot of different ways to play sounds. This format gives you access to all the sound playing tools available to you.
The system uses the X11 names for the keys. This allows for the use of special keys like F1, F2, F3, alt-A, alt-B, and so on.
If you are in sequential mode, the configuration files are seq_key.txt and seq_image.txt. These files contain a list of images (one per line) and commands(one per line).
Here is a sample seq_key.txt:
play words/alphab01.wav play words/boy00001.wav play words/colori06.wav ...
And here is a sample seq_image.txt:
jpeg/alphabet.jpeg jpeg/boy.jpeg jpeg/color.jpeg
Finally, to get out of the program, you need to type exit. (Four images will be displayed while you do this, but it does get you out.)
Clicking the close button does not close the application. Because the mouse has been grabbed, all mouse clicks go to the script and not the window manager.
The Results
When the program runs, it fills the screen with a picture and plays a sound. Here, you can see the result of a properly configured program after the C key has been pressed. (Pretend you're hearing mooing when you view this.)
One of the problems with designing configuration files for this program is that you don't necessarily know all the key names. After all, there are some awful strange key combinations out there. (What is the name of the key you get when you press alt, shift, ctrl, keypad dot?[*]) Every time the system sees a key with no image or sound, it writes a new entry to the file no_key.txt. Later you can use this file to design better configuration files.
[*] Because this program reads scan codes, you get four keys: alt_L, shift_L, ctrl_L, and KP_Decimal.
How It Works
The script is designed to completely take over the screen and the keyboard. After all, Grace isn't old enough to understand the concept of windows, much less how to manipulate them.
The script uses the Perl/Tk toolkit and creates a big top level window:
223 $mw = MainWindow->new(-title => "Grace's Program"); 224 225 # Big main window 226 my $big = $mw->Toplevel(); 227
Ideally, you would like one big borderless window to take over the whole screen. There is a Tk function to make the window borderless, but when I tried it, I couldn't get any key input. So I had to comment out this code until I can figure out how to make it work:
228 #
229 # Don't display borders
230 # (And don't work if commented in)
231 #
232 #if ($#ARGV == -1) {
233 # $big->overrideredirect(1);
234 #}
Next you get the height and width so that you can use it later when creating the Tk Canvas widget to hold the image. Then if you are debug mode, you shrink down the size of the window to make enough room on the screen for a debug window:
239 # Width and height of the screen
240 my $width = $mw->screenwidth();
241 my $height = $mw->screenheight();
242
243 if ($mode eq "debug") {
244 $width = 800;
245 $height = 600;
246 }
Now you create the canvas, which will cover the entire screen and be used for image display:
248 $canvas = $big->Canvas(-background => "Yellow", 249 -width => $width, 250 -height => $height 251 )->pack( 252 -expand => 1, 253 -fill => "both" 254 );
The script needs to handle all keyboard input. So you tell Perl/Tk to call the function key_handler any time a key is pressed:
236 $mw->bind("<KeyPress>" => &key_handler);
237 $big->bind("<KeyPress>" => &key_handler);
Finally, you grab the keyboard and mouse, which means that no other program can use them until the program releases its hold on them. This prevents Grace from typing things into other programs.
When Grace presses a key, the key_handler function is called. The first thing this function does is determine what key was pressed:
98 sub key_handler($) {
99 # Widget generating the event
100 my ($widget) = @_;
101
102 # The event causing the problem
103 my $event = $widget->XEvent;
104
105 # The key causing the event
106 my $key = $event->K();
Next you check to see if you are in the middle of typing exit to get out of the program:
108 if ($exit[$stage] eq $key) {
109 $stage++;
110 }
111 if ($stage > $#exit) {
112 exit (0);
113 }
The job of the program is to display an image and play a sound. The script now locates the image and sound for this key:
119 my $image_name = get_image($key); 120 my $sound = get_sound($key);
The image uses the Tk::Photo package:
125 if (defined($image_name)) {
126 # Define an image
127 my $image =
128 $mw->Photo(-file => $image_name);
129
130 if (defined($canvas_image)) {
131 $canvas->delete($canvas_image);
132 }
133 $canvas_image= $canvas->createImage(0, 0,
134 -anchor => "nw",
135 -image => $image);
136 }
You also fork off a process to run the command to play the sounds:
144 if (defined($sound)) {
145 if ($bg_pid == 0) {
146 $bg_pid = fork();
147 if ($bg_pid == 0) {
148 exec($sound);
149 }
150 }
151 }
Playing sounds in the background presents an interesting challenge. Suppose a long sound is playing in the background and Grace hits another key. What should you do?
The first version of this program tried to kill the background program and play the new sound. This didn't work well. One of the problems had to do with the design of the Linux play command. Killing this program does not release the sound device (that's a bug in play, not a problem with the script).
To work around this problem, the script was redesigned so that if it is playing a sound, it will ignore new keystrokes. When you play a sound, the PID (process ID) of the background process is stored in the variable $bg_pid.
If this variable is nonzero, then you have a background processing running and you ignore any new keystrokes:
114 # Lock system until bg sound finishes
115 if ($bg_pid != 0) {
116 return;
117 }
When the background process exits, the system generates a SIGCHLD. The script defines a handler for this signal:
205 $SIG{CHLD} = &child_handler;
When the child exists, the function is called. This function checks to make sure the exiting process is correct and clears the variable $bg_pid:
45 sub child_handler()
46 {
47 my $wait_pid = waitpid(-1, WNOHANG);
48 if ($wait_pid == $bg_pid) {
49 $bg_pid = 0;
50 }
51 }
This code does slow down the speed at which images can be displayed, but Grace doesn't care. She just bangs away at the keyboard and laughs.
Hacking the Script
I learned a lot writing this script. For example, I now know how to remove Play-Doh from a keyboard.
Also, I discovered that the grab function does not grab all the keys on the keyboard. On my laptop, there a big silver button labeled Power. Grace will hit that just as hard as she will any other key. Unfortunately, every time she hits it, the computer turns off.
Grace doesn't know how to talk yet, so she signals that she's done by throwing the keyboard to the ground. She's very good at throwing the keyboard down with enough force to pop a few keys off it. I'm getting very good at hunting for lost keys and popping them back on. (I'm typing this on a keyboard that's missing the * and - from the numeric pad.)
Currently the script ignores the mouse. It would be nice if the script would do something when a mouse button is clicked.
As it stands now, the script will serve Grace for the next six months or so. After that, we'll see what develops.