Introduction

Without concrete symbols, a computer is merely a pile of junk.

—Neil Postman

Understanding the basic concepts of character encoding is necessary for creating, manipulating, and rendering any kind of character data. Whenever data is brought into SAS from various external sources (external databases, flat files, etc.), whenever data is transferred between SAS applications running with different languages or across the network via thin clients, and whenever output is written to external files, SAS data sets, printers, or Web pages, an encoding is involved. In each of these cases, something can go wrong.

Though SAS has made it quite easy to run applications in many different languages, it is ultimately the user’s responsibility to ensure that their data is stored, processed, and rendered in the correct encoding.

If errors, notes, or warnings like the ones that follow are causing you nightmares, don’t worry. Things are not that bad. This book will help you to understand the fundamental concepts of characters, their representation in computer storage, character encoding standards and their implementation and usage in software applications. Last but not least, it gives practical examples for how to troubleshoot encoding problems.

ERROR: Some character data was lost during transcoding in the data set libref.member.

ERROR: Some code points did not transcode.

ERROR: File libref.member cannot be updated because its encoding does not match the session encoding or the file is in a format native to another host, such as WINDOWS_32.

NOTE: Data file libref.member is in a format native to another host or the file encoding does not match the session encoding. Cross Environment Data Access will be used, which may require additional CPU resources and reduce performance.

WARNING: A character that could not be transcoded was encountered.

The four main chapters are consecutive and based on each other.

Chapter 1 discusses different categories of writing systems and explains the basics of characters, character repertoires, glyphs, and fonts.

Chapter 2 discusses character encoding standards and explains the principles of transcoding.

Chapter 3 discusses the SAS approach to encodings.

Chapter 4 discusses how to troubleshoot encoding problems.

You can read through the book chapter by chapter, or you might want to read the troubleshooting chapter first, and then go back to get more information on the SAS approach to encodings and character encoding standards in general. The “References” section will point you to some further reading and give you access to online resources such as code charts and tutorials. Important terms and acronyms are explained in the text or in endnotes, and again in the “Glossary” section.

Now, I invite you to travel with me on a virtual journey through time and culture.