Preface

There are more things in heaven and earth, Horatio,
Than are dreamt of in your philosophy.

—Hamlet, Act I, scene V

This book is based on long-term experience with troubleshooting encoding problems, especially at SAS Institute Inc.

However, my actual first encounter with encodings was while working as a consultant for information retrieval systems at the German chemical company BASF. One of my jobs at that time was to administer a mainframe-based information retrieval system, customize it for the German language, and document the modifications for internal customers. At that time I wondered why German characters such as ä and ü displayed as curly braces {} on my IBM 3270 emulation under certain circumstances. A couple of years later, when working at SAS Headquarters in Europe, I created mailer tapes for different translated versions of SAS using an application on VM/CMS. By the way, a mailer tape is a tape, sent to customers, that can contain additional procedures, data sets, usage notes, and so on. In this case, the additions were translated catalogs and message files—those were the days. One of the things I needed to make sure of was that French, Italian, German, and Spanish characters displayed correctly when switching the terminal emulator settings. I then wondered why EBCDIC code pages often support the same character repertoire, but assign different code values to some of the characters.

There were even more mysteries to solve in the years to come, and often enough I have been clueless when confronted with a problem for the first time. Experience over many years helped, and I will try to share my experience with the readers of this book. Nevertheless, I need to clarify a couple of things first. This book is not meant to describe the latest and greatest features of SAS software. Due to the evolutionary nature of the software, there will always be enhancements and new and exciting features. This book is rather meant to explain things in a generic way, and first and foremost to help folks help themselves when it comes to finding answers to character encoding problems. It is also meant to give readers background information on characters and character encoding standards. The examples generally relate to SAS®9, and they are chosen so that they are of a general nature and are illustrated to show how to approach a problem or do an error analysis.

Why This Book Was Written

There are several good books on encodings (they are mentioned in the “References” section of this book) and on Unicode in particular, but none of them include the use of SAS software. Various SAS technical papers describe internationalization issues, but there is no how-to guide that gives people a broad overview of this important aspect of internationalization (and localization) of SAS software. This little book is supposed to fill that gap. Its main objective is to provide a good amount of information in compact form.

Although the discussion of encoding standards is intended to be of a general nature, it does draw primarily on the author’s experience at SAS Institute Inc. Still, the views expressed in this book are those of the author and not necessarily those of SAS.

The book is based on thorough research, but I have tried to write in simple, everyday language, without compromising its technical rigor. I leave it to the reader to decide whether I have succeeded.

Acknowledgments

Among the colleagues who reviewed the first draft and who encouraged me to write the book are Csaba Prencsovszky from SAS Hungary and Jean-Louis Dubois from SAS Belgium. My thanks also go to Shinichiro Kayano, who has been my long-term mentor on internationalization and all things Asian, and to Mr. Edwin Hart of the Johns Hopkins University Applied Physics Laboratory for his advice and information on the history of EBCDIC. At the risk of offending those I omit by mistake I acknowledge with gratitude the following reviewers (in alphabetical order): Junichi Arai, Maria Grazia Croatto, Bari Lawhorn, Masahiro Takagi, and Katherine Underwood.

My special thanks go to my colleague from the SAS Globalization Division, Elizabeth Bales, for her extremely valuable feedback, which helped me to weed out several errors or inconsistencies in the author’s manuscript before it was published. Any remaining errors in the book are entirely mine. Additionally, I want to thank my manager, Steve Beatrous, for supporting the creation of this book.

Finally, none of this would have been possible without the patience and dedication of the staff of SAS Press. In particular I would like to thank George McDaniel and Shelley Sessoms as representatives for all others.