The GNU C Library

Short Contents

Table of Contents

Next: , Previous: (dir), Up: (dir)

Main Menu

This is Edition 0.11, last updated 2007-09-09, of The GNU C Library Reference Manual, for Version 2.7 of the GNU C Library.

Appendices

Indices

--- The Detailed Node Listing ---

Introduction

Standards and Portability

Using the Library

Error Reporting

Memory

Memory Allocation

Unconstrained Allocation

Allocation Debugging

Obstacks

Variable Size Automatic

Locking Pages

Character Handling

String and Array Utilities

Argz and Envz Vectors

Character Set Handling

Restartable multibyte conversion

Non-reentrant Conversion

Generic Charset Conversion

Locales

Locale Information

The Lame Way to Locale Data

Message Translation

Message catalogs a la X/Open

The Uniforum approach

Message catalogs with gettext

Searching and Sorting

Pattern Matching

Globbing

Regular Expressions

Word Expansion

I/O Overview

I/O Concepts

File Names

I/O on Streams

Unreading

Formatted Output

Customizing Printf

Formatted Input

Stream Buffering

Other Kinds of Streams

Custom Streams

Formatted Messages

Low-Level I/O

Stream/Descriptor Precautions

Asynchronous I/O

File Status Flags

File System Interface

Accessing Directories

File Attributes

Pipes and FIFOs

Sockets

Socket Addresses

Local Namespace

Internet Namespace

Host Addresses

Open/Close Sockets

Connections

Transferring Data

Datagrams

Inetd

Socket Options

Low-Level Terminal Interface

Terminal Modes

Special Characters

Pseudo-Terminals

Syslog

Submitting Syslog Messages

Mathematics

Pseudo-Random Numbers

Arithmetic

Floating Point Errors

Arithmetic Functions

Parsing of Numbers

Date and Time

Processor And CPU Time

Calendar Time

Parsing Date and Time

Resource Usage And Limitation

Priority

Traditional Scheduling

Memory Resources

Non-Local Exits

Signal Handling

Concepts of Signals

Standard Signals

Signal Actions

Defining Handlers

Atomic Data Access

Generating Signals

Blocking Signals

Waiting for a Signal

BSD Signal Handling

Program Basics

Program Arguments

Parsing Program Arguments

Environment Variables

Program Termination

Processes

Job Control

Implementing a Shell

Functions for Job Control

Name Service Switch

NSS Configuration File

NSS Module Internals

Extending NSS

Users and Groups

User Accounting Database

User Database

Group Database

Netgroup Database

System Management

Filesystem Handling

Mount Information

System Configuration

Sysconf

Cryptographic Functions

Debugging Support

Language Features

Variadic Functions

How Variadic

Data Type Measurements

Floating Type Macros

Installation

Maintenance

Porting

Next: , Previous: Top, Up: Top

1 Introduction

The C language provides no built-in facilities for performing such common operations as input/output, memory management, string manipulation, and the like. Instead, these facilities are defined in a standard library, which you compile and link with your programs. The GNU C library, described in this document, defines all of the library functions that are specified by the ISO C standard, as well as additional features specific to POSIX and other derivatives of the Unix operating system, and extensions specific to the GNU system.

The purpose of this manual is to tell you how to use the facilities of the GNU library. We have mentioned which features belong to which standards to help you identify things that are potentially non-portable to other systems. But the emphasis in this manual is not on strict portability.

Next: , Up: Introduction

1.1 Getting Started

This manual is written with the assumption that you are at least somewhat familiar with the C programming language and basic programming concepts. Specifically, familiarity with ISO standard C (see ISO C), rather than “traditional” pre-ISO C dialects, is assumed.

The GNU C library includes several header files, each of which provides definitions and declarations for a group of related facilities; this information is used by the C compiler when processing your program. For example, the header file stdio.h declares facilities for performing input and output, and the header file string.h declares string processing utilities. The organization of this manual generally follows the same division as the header files.

If you are reading this manual for the first time, you should read all of the introductory material and skim the remaining chapters. There are a lot of functions in the GNU C library and it's not realistic to expect that you will be able to remember exactly how to use each and every one of them. It's more important to become generally familiar with the kinds of facilities that the library provides, so that when you are writing your programs you can recognize when to make use of library functions, and where in this manual you can find more specific information about them.

Next: , Previous: Getting Started, Up: Introduction

1.2 Standards and Portability

This section discusses the various standards and other sources that the GNU C library is based upon. These sources include the ISO C and POSIX standards, and the System V and Berkeley Unix implementations.

The primary focus of this manual is to tell you how to make effective use of the GNU library facilities. But if you are concerned about making your programs compatible with these standards, or portable to operating systems other than GNU, this can affect how you use the library. This section gives you an overview of these standards, so that you will know what they are when they are mentioned in other parts of the manual.

See Library Summary, for an alphabetical list of the functions and other symbols provided by the library. This list also states which standards each function or symbol comes from.

Next: , Up: Standards and Portability

1.2.1 ISO C

The GNU C library is compatible with the C standard adopted by the American National Standards Institute (ANSI): American National Standard X3.159-1989—“ANSI C” and later by the International Standardization Organization (ISO): ISO/IEC 9899:1990, “Programming languages—C”. We here refer to the standard as ISO C since this is the more general standard in respect of ratification. The header files and library facilities that make up the GNU library are a superset of those specified by the ISO C standard.

If you are concerned about strict adherence to the ISO C standard, you should use the `-ansi' option when you compile your programs with the GNU C compiler. This tells the compiler to define only ISO standard features from the library header files, unless you explicitly ask for additional features. See Feature Test Macros, for information on how to do this.

Being able to restrict the library to include only ISO C features is important because ISO C puts limitations on what names can be defined by the library implementation, and the GNU extensions don't fit these limitations. See Reserved Names, for more information about these restrictions.

This manual does not attempt to give you complete details on the differences between ISO C and older dialects. It gives advice on how to write programs to work portably under multiple C dialects, but does not aim for completeness.

Next: , Previous: ISO C, Up: Standards and Portability

1.2.2 POSIX (The Portable Operating System Interface)

The GNU library is also compatible with the ISO POSIX family of standards, known more formally as the Portable Operating System Interface for Computer Environments (ISO/IEC 9945). They were also published as ANSI/IEEE Std 1003. POSIX is derived mostly from various versions of the Unix operating system.

The library facilities specified by the POSIX standards are a superset of those required by ISO C; POSIX specifies additional features for ISO C functions, as well as specifying new additional functions. In general, the additional requirements and functionality defined by the POSIX standards are aimed at providing lower-level support for a particular kind of operating system environment, rather than general programming language support which can run in many diverse operating system environments.

The GNU C library implements all of the functions specified in ISO/IEC 9945-1:1996, the POSIX System Application Program Interface, commonly referred to as POSIX.1. The primary extensions to the ISO C facilities specified by this standard include file system interface primitives (see File System Interface), device-specific terminal control functions (see Low-Level Terminal Interface), and process control functions (see Processes).

Some facilities from ISO/IEC 9945-2:1993, the POSIX Shell and Utilities standard (POSIX.2) are also implemented in the GNU library. These include utilities for dealing with regular expressions and other pattern matching facilities (see Pattern Matching).

Next: , Previous: POSIX, Up: Standards and Portability

1.2.3 Berkeley Unix

The GNU C library defines facilities from some versions of Unix which are not formally standardized, specifically from the 4.2 BSD, 4.3 BSD, and 4.4 BSD Unix systems (also known as Berkeley Unix) and from SunOS (a popular 4.2 BSD derivative that includes some Unix System V functionality). These systems support most of the ISO C and POSIX facilities, and 4.4 BSD and newer releases of SunOS in fact support them all.

The BSD facilities include symbolic links (see Symbolic Links), the select function (see Waiting for I/O), the BSD signal functions (see BSD Signal Handling), and sockets (see Sockets).

Next: , Previous: Berkeley Unix, Up: Standards and Portability

1.2.4 SVID (The System V Interface Description)

The System V Interface Description (SVID) is a document describing the AT&T Unix System V operating system. It is to some extent a superset of the POSIX standard (see POSIX).

The GNU C library defines most of the facilities required by the SVID that are not also required by the ISO C or POSIX standards, for compatibility with System V Unix and other Unix systems (such as SunOS) which include these facilities. However, many of the more obscure and less generally useful facilities required by the SVID are not included. (In fact, Unix System V itself does not provide them all.)

The supported facilities from System V include the methods for inter-process communication and shared memory, the hsearch and drand48 families of functions, fmtmsg and several of the mathematical functions.

Previous: SVID, Up: Standards and Portability

1.2.5 XPG (The X/Open Portability Guide)

The X/Open Portability Guide, published by the X/Open Company, Ltd., is a more general standard than POSIX. X/Open owns the Unix copyright and the XPG specifies the requirements for systems which are intended to be a Unix system.

The GNU C library complies to the X/Open Portability Guide, Issue 4.2, with all extensions common to XSI (X/Open System Interface) compliant systems and also all X/Open UNIX extensions.

The additions on top of POSIX are mainly derived from functionality available in System V and BSD systems. Some of the really bad mistakes in System V systems were corrected, though. Since fulfilling the XPG standard with the Unix extensions is a precondition for getting the Unix brand chances are good that the functionality is available on commercial systems.

Next: , Previous: Standards and Portability, Up: Introduction

1.3 Using the Library

This section describes some of the practical issues involved in using the GNU C library.

Next: , Up: Using the Library

1.3.1 Header Files

Libraries for use by C programs really consist of two parts: header files that define types and macros and declare variables and functions; and the actual library or archive that contains the definitions of the variables and functions.

(Recall that in C, a declaration merely provides information that a function or variable exists and gives its type. For a function declaration, information about the types of its arguments might be provided as well. The purpose of declarations is to allow the compiler to correctly process references to the declared variables and functions. A definition, on the other hand, actually allocates storage for a variable or says what a function does.) In order to use the facilities in the GNU C library, you should be sure that your program source files include the appropriate header files. This is so that the compiler has declarations of these facilities available and can correctly process references to them. Once your program has been compiled, the linker resolves these references to the actual definitions provided in the archive file.

Header files are included into a program source file by the `#include' preprocessor directive. The C language supports two forms of this directive; the first, 

     #include "header"

is typically used to include a header file header that you write yourself; this would contain definitions and declarations describing the interfaces between the different parts of your particular application. By contrast, 

     #include <file.h>

is typically used to include a header file file.h that contains definitions and declarations for a standard library. This file would normally be installed in a standard place by your system administrator. You should use this second form for the C library header files.

Typically, `#include' directives are placed at the top of the C source file, before any other code. If you begin your source files with some comments explaining what the code in the file does (a good idea), put the `#include' directives immediately afterwards, following the feature test macro definition (see Feature Test Macros).

For more information about the use of header files and `#include' directives, see Header Files.

The GNU C library provides several header files, each of which contains the type and macro definitions and variable and function declarations for a group of related facilities. This means that your programs may need to include several header files, depending on exactly which facilities you are using.

Some library header files include other library header files automatically. However, as a matter of programming style, you should not rely on this; it is better to explicitly include all the header files required for the library facilities you are using. The GNU C library header files have been written in such a way that it doesn't matter if a header file is accidentally included more than once; including a header file a second time has no effect. Likewise, if your program needs to include multiple header files, the order in which they are included doesn't matter.

Compatibility Note: Inclusion of standard header files in any order and any number of times works in any ISO C implementation. However, this has traditionally not been the case in many older C implementations.

Strictly speaking, you don't have to include a header file to use a function it declares; you could declare the function explicitly yourself, according to the specifications in this manual. But it is usually better to include the header file because it may define types and macros that are not otherwise available and because it may define more efficient macro replacements for some functions. It is also a sure way to have the correct declaration.

Next: , Previous: Header Files, Up: Using the Library

1.3.2 Macro Definitions of Functions

If we describe something as a function in this manual, it may have a macro definition as well. This normally has no effect on how your program runs—the macro definition does the same thing as the function would. In particular, macro equivalents for library functions evaluate arguments exactly once, in the same way that a function call would. The main reason for these macro definitions is that sometimes they can produce an inline expansion that is considerably faster than an actual function call.

Taking the address of a library function works even if it is also defined as a macro. This is because, in this context, the name of the function isn't followed by the left parenthesis that is syntactically necessary to recognize a macro call.

You might occasionally want to avoid using the macro definition of a function—perhaps to make your program easier to debug. There are two ways you can do this:

For example, suppose the header file stdlib.h declares a function named abs with 

     extern int abs (int);

and also provides a macro definition for abs. Then, in: 

     #include <stdlib.h>
     int f (int *i) { return abs (++*i); }

the reference to abs might refer to either a macro or a function. On the other hand, in each of the following examples the reference is to a function and not a macro. 

     #include <stdlib.h>
     int g (int *i) { return (abs) (++*i); }
     
     #undef abs
     int h (int *i) { return abs (++*i); }

Since macro definitions that double for a function behave in exactly the same way as the actual function version, there is usually no need for any of these methods. In fact, removing macro definitions usually just makes your program slower.

Next: , Previous: Macro Definitions, Up: Using the Library

1.3.3 Reserved Names

The names of all library types, macros, variables and functions that come from the ISO C standard are reserved unconditionally; your program may not redefine these names. All other library names are reserved if your program explicitly includes the header file that defines or declares them. There are several reasons for these restrictions:

In addition to the names documented in this manual, reserved names include all external identifiers (global functions and variables) that begin with an underscore (`_') and all identifiers regardless of use that begin with either two underscores or an underscore followed by a capital letter are reserved names. This is so that the library and header files can define functions, variables, and macros for internal purposes without risk of conflict with names in user programs.

Some additional classes of identifier names are reserved for future extensions to the C language or the POSIX.1 environment. While using these names for your own purposes right now might not cause a problem, they do raise the possibility of conflict with future versions of the C or POSIX standards, so you should avoid these names.

In addition, some individual header files reserve names beyond those that they actually define. You only need to worry about these restrictions if your program includes that particular header file.

Previous: Reserved Names, Up: Using the Library

1.3.4 Feature Test Macros

The exact set of features available when you compile a source file is controlled by which feature test macros you define.

If you compile your programs using `gcc -ansi', you get only the ISO C library features, unless you explicitly request additional features by defining one or more of the feature macros. See GNU CC Command Options, for more information about GCC options.

You should define these macros by using `#define' preprocessor directives at the top of your source code files. These directives must come before any #include of a system header file. It is best to make them the very first thing in the file, preceded only by comments. You could also use the `-D' option to GCC, but it's better if you make the source files indicate their own meaning in a self-contained way.

This system exists to allow the library to conform to multiple standards. Although the different standards are often described as supersets of each other, they are usually incompatible because larger standards require functions with names that smaller ones reserve to the user program. This is not mere pedantry — it has been a problem in practice. For instance, some non-GNU programs define functions named getline that have nothing to do with this library's getline. They would not be compilable if all features were enabled indiscriminately.

This should not be used to verify that a program conforms to a limited standard. It is insufficient for this purpose, as it will not protect you from including header files outside the standard, or relying on semantics undefined within the standard.

— Macro: _POSIX_SOURCE

If you define this macro, then the functionality from the POSIX.1 standard (IEEE Standard 1003.1) is available, as well as all of the ISO C facilities.

The state of _POSIX_SOURCE is irrelevant if you define the macro _POSIX_C_SOURCE to a positive integer.

— Macro: _POSIX_C_SOURCE

Define this macro to a positive integer to control which POSIX functionality is made available. The greater the value of this macro, the more functionality is made available.

If you define this macro to a value greater than or equal to 1, then the functionality from the 1990 edition of the POSIX.1 standard (IEEE Standard 1003.1-1990) is made available.

If you define this macro to a value greater than or equal to 2, then the functionality from the 1992 edition of the POSIX.2 standard (IEEE Standard 1003.2-1992) is made available.

If you define this macro to a value greater than or equal to 199309L, then the functionality from the 1993 edition of the POSIX.1b standard (IEEE Standard 1003.1b-1993) is made available.

Greater values for _POSIX_C_SOURCE will enable future extensions. The POSIX standards process will define these values as necessary, and the GNU C Library should support them some time after they become standardized. The 1996 edition of POSIX.1 (ISO/IEC 9945-1: 1996) states that if you define _POSIX_C_SOURCE to a value greater than or equal to 199506L, then the functionality from the 1996 edition is made available.

— Macro: _BSD_SOURCE

If you define this macro, functionality derived from 4.3 BSD Unix is included as well as the ISO C, POSIX.1, and POSIX.2 material.

Some of the features derived from 4.3 BSD Unix conflict with the corresponding features specified by the POSIX.1 standard. If this macro is defined, the 4.3 BSD definitions take precedence over the POSIX definitions.

Due to the nature of some of the conflicts between 4.3 BSD and POSIX.1, you need to use a special BSD compatibility library when linking programs compiled for BSD compatibility. This is because some functions must be defined in two different ways, one of them in the normal C library, and one of them in the compatibility library. If your program defines _BSD_SOURCE, you must give the option `-lbsd-compat' to the compiler or linker when linking the program, to tell it to find functions in this special compatibility library before looking for them in the normal C library.

— Macro: _SVID_SOURCE

If you define this macro, functionality derived from SVID is included as well as the ISO C, POSIX.1, POSIX.2, and X/Open material.

— Macro: _XOPEN_SOURCE
— Macro: _XOPEN_SOURCE_EXTENDED

If you define this macro, functionality described in the X/Open Portability Guide is included. This is a superset of the POSIX.1 and POSIX.2 functionality and in fact _POSIX_SOURCE and _POSIX_C_SOURCE are automatically defined.

As the unification of all Unices, functionality only available in BSD and SVID is also included.

If the macro _XOPEN_SOURCE_EXTENDED is also defined, even more functionality is available. The extra functions will make all functions available which are necessary for the X/Open Unix brand.

If the macro _XOPEN_SOURCE has the value 500 this includes all functionality described so far plus some new definitions from the Single Unix Specification, version 2.

— Macro: _LARGEFILE_SOURCE

If this macro is defined some extra functions are available which rectify a few shortcomings in all previous standards. Specifically, the functions fseeko and ftello are available. Without these functions the difference between the ISO C interface (fseek, ftell) and the low-level POSIX interface (lseek) would lead to problems.

This macro was introduced as part of the Large File Support extension (LFS).

— Macro: _LARGEFILE64_SOURCE

If you define this macro an additional set of functions is made available which enables 32 bit systems to use files of sizes beyond the usual limit of 2GB. This interface is not available if the system does not support files that large. On systems where the natural file size limit is greater than 2GB (i.e., on 64 bit systems) the new functions are identical to the replaced functions.

The new functionality is made available by a new set of types and functions which replace the existing ones. The names of these new objects contain 64 to indicate the intention, e.g., off_t vs. off64_t and fseeko vs. fseeko64.

This macro was introduced as part of the Large File Support extension (LFS). It is a transition interface for the period when 64 bit offsets are not generally used (see _FILE_OFFSET_BITS).

— Macro: _FILE_OFFSET_BITS

This macro determines which file system interface shall be used, one replacing the other. Whereas _LARGEFILE64_SOURCE makes the 64 bit interface available as an additional interface, _FILE_OFFSET_BITS allows the 64 bit interface to replace the old interface.

If _FILE_OFFSET_BITS is undefined, or if it is defined to the value 32, nothing changes. The 32 bit interface is used and types like off_t have a size of 32 bits on 32 bit systems.

If the macro is defined to the value 64, the large file interface replaces the old interface. I.e., the functions are not made available under different names (as they are with _LARGEFILE64_SOURCE). Instead the old function names now reference the new functions, e.g., a call to fseeko now indeed calls fseeko64.

This macro should only be selected if the system provides mechanisms for handling large files. On 64 bit systems this macro has no effect since the *64 functions are identical to the normal functions.

This macro was introduced as part of the Large File Support extension (LFS).

— Macro: _ISOC99_SOURCE

Until the revised ISO C standard is widely adopted the new features are not automatically enabled. The GNU libc nevertheless has a complete implementation of the new standard and to enable the new features the macro _ISOC99_SOURCE should be defined.

— Macro: _GNU_SOURCE

If you define this macro, everything is included: ISO C89, ISO C99, POSIX.1, POSIX.2, BSD, SVID, X/Open, LFS, and GNU extensions. In the cases where POSIX.1 conflicts with BSD, the POSIX definitions take precedence.

If you want to get the full effect of _GNU_SOURCE but make the BSD definitions take precedence over the POSIX definitions, use this sequence of definitions: 

          #define _GNU_SOURCE
          #define _BSD_SOURCE
          #define _SVID_SOURCE

Note that if you do this, you must link your program with the BSD compatibility library by passing the `-lbsd-compat' option to the compiler or linker. Note: If you forget to do this, you may get very strange errors at run time.

— Macro: _REENTRANT
— Macro: _THREAD_SAFE

If you define one of these macros, reentrant versions of several functions get declared. Some of the functions are specified in POSIX.1c but many others are only available on a few other systems or are unique to GNU libc. The problem is the delay in the standardization of the thread safe C library interface.

Unlike on some other systems, no special version of the C library must be used for linking. There is only one version but while compiling this it must have been specified to compile as thread safe.

We recommend you use _GNU_SOURCE in new programs. If you don't specify the `-ansi' option to GCC and don't define any of these macros explicitly, the effect is the same as defining _POSIX_C_SOURCE to 2 and _POSIX_SOURCE, _SVID_SOURCE, and _BSD_SOURCE to 1.

When you define a feature test macro to request a larger class of features, it is harmless to define in addition a feature test macro for a subset of those features. For example, if you define _POSIX_C_SOURCE, then defining _POSIX_SOURCE as well has no effect. Likewise, if you define _GNU_SOURCE, then defining either _POSIX_SOURCE or _POSIX_C_SOURCE or _SVID_SOURCE as well has no effect.

Note, however, that the features of _BSD_SOURCE are not a subset of any of the other feature test macros supported. This is because it defines BSD features that take precedence over the POSIX features that are requested by the other macros. For this reason, defining _BSD_SOURCE in addition to the other feature test macros does have an effect: it causes the BSD features to take priority over the conflicting POSIX features.

Previous: Using the Library, Up: Introduction

1.4 Roadmap to the Manual

Here is an overview of the contents of the remaining chapters of this manual.

If you already know the name of the facility you are interested in, you can look it up in Library Summary. This gives you a summary of its syntax and a pointer to where you can find a more detailed description. This appendix is particularly useful if you just want to verify the order and type of arguments to a function, for example. It also tells you what standard or system each function, variable, or macro is derived from.

Next: , Previous: Introduction, Up: Top

2 Error Reporting

Many functions in the GNU C library detect and report error conditions, and sometimes your programs need to check for these error conditions. For example, when you open an input file, you should verify that the file was actually opened correctly, and print an error message or take other appropriate action if the call to the library function failed.

This chapter describes how the error reporting facility works. Your program should include the header file errno.h to use this facility.

Next: , Up: Error Reporting

2.1 Checking for Errors

Most library functions return a special value to indicate that they have failed. The special value is typically -1, a null pointer, or a constant such as EOF that is defined for that purpose. But this return value tells you only that an error has occurred. To find out what kind of error it was, you need to look at the error code stored in the variable errno. This variable is declared in the header file errno.h.

— Variable: volatile int errno

The variable errno contains the system error number. You can change the value of errno.

Since errno is declared volatile, it might be changed asynchronously by a signal handler; see Defining Handlers. However, a properly written signal handler saves and restores the value of errno, so you generally do not need to worry about this possibility except when writing signal handlers.

The initial value of errno at program startup is zero. Many library functions are guaranteed to set it to certain nonzero values when they encounter certain kinds of errors. These error conditions are listed for each function. These functions do not change errno when they succeed; thus, the value of errno after a successful call is not necessarily zero, and you should not use errno to determine whether a call failed. The proper way to do that is documented for each function. If the call failed, you can examine errno.

Many library functions can set errno to a nonzero value as a result of calling other library functions which might fail. You should assume that any library function might alter errno when the function returns an error.

Portability Note: ISO C specifies errno as a “modifiable lvalue” rather than as a variable, permitting it to be implemented as a macro. For example, its expansion might involve a function call, like *_errno (). In fact, that is what it is on the GNU system itself. The GNU library, on non-GNU systems, does whatever is right for the particular system.

There are a few library functions, like sqrt and atan, that return a perfectly legitimate value in case of an error, but also set errno. For these functions, if you want to check to see whether an error occurred, the recommended method is to set errno to zero before calling the function, and then check its value afterward.

All the error codes have symbolic names; they are macros defined in errno.h. The names start with `E' and an upper-case letter or digit; you should consider names of this form to be reserved names. See Reserved Names.

The error code values are all positive integers and are all distinct, with one exception: EWOULDBLOCK and EAGAIN are the same. Since the values are distinct, you can use them as labels in a switch statement; just don't use both EWOULDBLOCK and EAGAIN. Your program should not make any other assumptions about the specific values of these symbolic constants.

The value of errno doesn't necessarily have to correspond to any of these macros, since some library functions might return other error codes of their own for other situations. The only values that are guaranteed to be meaningful for a particular library function are the ones that this manual lists for that function.

On non-GNU systems, almost any system call can return EFAULT if it is given an invalid pointer as an argument. Since this could only happen as a result of a bug in your program, and since it will not happen on the GNU system, we have saved space by not mentioning EFAULT in the descriptions of individual functions.

In some Unix systems, many system calls can also return EFAULT if given as an argument a pointer into the stack, and the kernel for some obscure reason fails in its attempt to extend the stack. If this ever happens, you should probably try using statically or dynamically allocated memory instead of stack memory on that system.

Next: , Previous: Checking for Errors, Up: Error Reporting

2.2 Error Codes

The error code macros are defined in the header file errno.h. All of them expand into integer constant values. Some of these error codes can't occur on the GNU system, but they can occur using the GNU library on other systems.

— Macro: int EPERM

Operation not permitted; only the owner of the file (or other resource) or processes with special privileges can perform the operation.

— Macro: int ENOENT

No such file or directory. This is a “file doesn't exist” error for ordinary files that are referenced in contexts where they are expected to already exist.

— Macro: int ESRCH

No process matches the specified process ID.

— Macro: int EINTR

Interrupted function call; an asynchronous signal occurred and prevented completion of the call. When this happens, you should try the call again.

You can choose to have functions resume after a signal that is handled, rather than failing with EINTR; see Interrupted Primitives.

— Macro: int EIO

Input/output error; usually used for physical read or write errors.

— Macro: int ENXIO

No such device or address. The system tried to use the device represented by a file you specified, and it couldn't find the device. This can mean that the device file was installed incorrectly, or that the physical device is missing or not correctly attached to the computer.

— Macro: int E2BIG

Argument list too long; used when the arguments passed to a new program being executed with one of the exec functions (see Executing a File) occupy too much memory space. This condition never arises in the GNU system.

— Macro: int ENOEXEC

Invalid executable file format. This condition is detected by the exec functions; see Executing a File.

— Macro: int EBADF

Bad file descriptor; for example, I/O on a descriptor that has been closed or reading from a descriptor open only for writing (or vice versa).

— Macro: int ECHILD

There are no child processes. This error happens on operations that are supposed to manipulate child processes, when there aren't any processes to manipulate.

— Macro: int EDEADLK

Deadlock avoided; allocating a system resource would have resulted in a deadlock situation. The system does not guarantee that it will notice all such situations. This error means you got lucky and the system noticed; it might just hang. See File Locks, for an example.

— Macro: int ENOMEM

No memory available. The system cannot allocate more virtual memory because its capacity is full.

— Macro: int EACCES

Permission denied; the file permissions do not allow the attempted operation.

— Macro: int EFAULT

Bad address; an invalid pointer was detected. In the GNU system, this error never happens; you get a signal instead.

— Macro: int ENOTBLK

A file that isn't a block special file was given in a situation that requires one. For example, trying to mount an ordinary file as a file system in Unix gives this error.

— Macro: int EBUSY

Resource busy; a system resource that can't be shared is already in use. For example, if you try to delete a file that is the root of a currently mounted filesystem, you get this error.

— Macro: int EEXIST

File exists; an existing file was specified in a context where it only makes sense to specify a new file.

— Macro: int EXDEV

An attempt to make an improper link across file systems was detected. This happens not only when you use link (see Hard Links) but also when you rename a file with rename (see Renaming Files).

— Macro: int ENODEV

The wrong type of device was given to a function that expects a particular sort of device.

— Macro: int ENOTDIR

A file that isn't a directory was specified when a directory is required.

— Macro: int EISDIR

File is a directory; you cannot open a directory for writing, or create or remove hard links to it.

— Macro: int EINVAL

Invalid argument. This is used to indicate various kinds of problems with passing the wrong argument to a library function.

— Macro: int EMFILE

The current process has too many files open and can't open any more. Duplicate descriptors do count toward this limit.

In BSD and GNU, the number of open files is controlled by a resource limit that can usually be increased. If you get this error, you might want to increase the RLIMIT_NOFILE limit or make it unlimited; see Limits on Resources.

— Macro: int ENFILE

There are too many distinct file openings in the entire system. Note that any number of linked channels count as just one file opening; see Linked Channels. This error never occurs in the GNU system.

— Macro: int ENOTTY

Inappropriate I/O control operation, such as trying to set terminal modes on an ordinary file.

— Macro: int ETXTBSY

An attempt to execute a file that is currently open for writing, or write to a file that is currently being executed. Often using a debugger to run a program is considered having it open for writing and will cause this error. (The name stands for “text file busy”.) This is not an error in the GNU system; the text is copied as necessary.

— Macro: int EFBIG

File too big; the size of a file would be larger than allowed by the system.

— Macro: int ENOSPC

No space left on device; write operation on a file failed because the disk is full.

— Macro: int ESPIPE

Invalid seek operation (such as on a pipe).

— Macro: int EROFS

An attempt was made to modify something on a read-only file system.

— Macro: int EMLINK

Too many links; the link count of a single file would become too large. rename can cause this error if the file being renamed already has as many links as it can take (see Renaming Files).

— Macro: int EPIPE

Broken pipe; there is no process reading from the other end of a pipe. Every library function that returns this error code also generates a SIGPIPE signal; this signal terminates the program if not handled or blocked. Thus, your program will never actually see EPIPE unless it has handled or blocked SIGPIPE.

— Macro: int EDOM

Domain error; used by mathematical functions when an argument value does not fall into the domain over which the function is defined.

— Macro: int ERANGE

Range error; used by mathematical functions when the result value is not representable because of overflow or underflow.

— Macro: int EAGAIN

Resource temporarily unavailable; the call might work if you try again later. The macro EWOULDBLOCK is another name for EAGAIN; they are always the same in the GNU C library.

This error can happen in a few different situations:

— Macro: int EWOULDBLOCK

In the GNU C library, this is another name for EAGAIN (above). The values are always the same, on every operating system.

C libraries in many older Unix systems have EWOULDBLOCK as a separate error code.

— Macro: int EINPROGRESS

An operation that cannot complete immediately was initiated on an object that has non-blocking mode selected. Some functions that must always block (such as connect; see Connecting) never return EAGAIN. Instead, they return EINPROGRESS to indicate that the operation has begun and will take some time. Attempts to manipulate the object before the call completes return EALREADY. You can use the select function to find out when the pending operation has completed; see Waiting for I/O.

— Macro: int EALREADY

An operation is already in progress on an object that has non-blocking mode selected.

— Macro: int ENOTSOCK

A file that isn't a socket was specified when a socket is required.

— Macro: int EMSGSIZE

The size of a message sent on a socket was larger than the supported maximum size.

— Macro: int EPROTOTYPE

The socket type does not support the requested communications protocol.

— Macro: int ENOPROTOOPT

You specified a socket option that doesn't make sense for the particular protocol being used by the socket. See Socket Options.

— Macro: int EPROTONOSUPPORT

The socket domain does not support the requested communications protocol (perhaps because the requested protocol is completely invalid). See Creating a Socket.

— Macro: int ESOCKTNOSUPPORT

The socket type is not supported.

— Macro: int EOPNOTSUPP

The operation you requested is not supported. Some socket functions don't make sense for all types of sockets, and others may not be implemented for all communications protocols. In the GNU system, this error can happen for many calls when the object does not support the particular operation; it is a generic indication that the server knows nothing to do for that call.

— Macro: int EPFNOSUPPORT

The socket communications protocol family you requested is not supported.

— Macro: int EAFNOSUPPORT

The address family specified for a socket is not supported; it is inconsistent with the protocol being used on the socket. See Sockets.

— Macro: int EADDRINUSE

The requested socket address is already in use. See Socket Addresses.

— Macro: int EADDRNOTAVAIL

The requested socket address is not available; for example, you tried to give a socket a name that doesn't match the local host name. See Socket Addresses.

— Macro: int ENETDOWN

A socket operation failed because the network was down.

— Macro: int ENETUNREACH

A socket operation failed because the subnet containing the remote host was unreachable.

— Macro: int ENETRESET

A network connection was reset because the remote host crashed.

— Macro: int ECONNABORTED

A network connection was aborted locally.

— Macro: int ECONNRESET

A network connection was closed for reasons outside the control of the local host, such as by the remote machine rebooting or an unrecoverable protocol violation.

— Macro: int ENOBUFS

The kernel's buffers for I/O operations are all in use. In GNU, this error is always synonymous with ENOMEM; you may get one or the other from network operations.

— Macro: int EISCONN

You tried to connect a socket that is already connected. See Connecting.

— Macro: int ENOTCONN

The socket is not connected to anything. You get this error when you try to transmit data over a socket, without first specifying a destination for the data. For a connectionless socket (for datagram protocols, such as UDP), you get EDESTADDRREQ instead.

— Macro: int EDESTADDRREQ

No default destination address was set for the socket. You get this error when you try to transmit data over a connectionless socket, without first specifying a destination for the data with connect.

— Macro: int ESHUTDOWN

The socket has already been shut down.

— Macro: int ETOOMANYREFS

???

— Macro: int ETIMEDOUT

A socket operation with a specified timeout received no response during the timeout period.

— Macro: int ECONNREFUSED

A remote host refused to allow the network connection (typically because it is not running the requested service).

— Macro: int ELOOP

Too many levels of symbolic links were encountered in looking up a file name. This often indicates a cycle of symbolic links.

— Macro: int ENAMETOOLONG

Filename too long (longer than PATH_MAX; see Limits for Files) or host name too long (in gethostname or sethostname; see Host Identification).

— Macro: int EHOSTDOWN

The remote host for a requested network connection is down.

— Macro: int EHOSTUNREACH

The remote host for a requested network connection is not reachable.

— Macro: int ENOTEMPTY

Directory not empty, where an empty directory was expected. Typically, this error occurs when you are trying to delete a directory.

— Macro: int EPROCLIM

This means that the per-user limit on new process would be exceeded by an attempted fork. See Limits on Resources, for details on the RLIMIT_NPROC limit.

— Macro: int EUSERS

The file quota system is confused because there are too many users.

— Macro: int EDQUOT

The user's disk quota was exceeded.

— Macro: int ESTALE

Stale NFS file handle. This indicates an internal confusion in the NFS system which is due to file system rearrangements on the server host. Repairing this condition usually requires unmounting and remounting the NFS file system on the local host.

— Macro: int EREMOTE

An attempt was made to NFS-mount a remote file system with a file name that already specifies an NFS-mounted file. (This is an error on some operating systems, but we expect it to work properly on the GNU system, making this error code impossible.)

— Macro: int EBADRPC

???

— Macro: int ERPCMISMATCH

???

— Macro: int EPROGUNAVAIL

???

— Macro: int EPROGMISMATCH

???

— Macro: int EPROCUNAVAIL

???

— Macro: int ENOLCK

No locks available. This is used by the file locking facilities; see File Locks. This error is never generated by the GNU system, but it can result from an operation to an NFS server running another operating system.

— Macro: int EFTYPE

Inappropriate file type or format. The file was the wrong type for the operation, or a data file had the wrong format.

On some systems chmod returns this error if you try to set the sticky bit on a non-directory file; see Setting Permissions.

— Macro: int EAUTH

???

— Macro: int ENEEDAUTH

???

— Macro: int ENOSYS

Function not implemented. This indicates that the function called is not implemented at all, either in the C library itself or in the operating system. When you get this error, you can be sure that this particular function will always fail with ENOSYS unless you install a new version of the C library or the operating system.

— Macro: int ENOTSUP

Not supported. A function returns this error when certain parameter values are valid, but the functionality they request is not available. This can mean that the function does not implement a particular command or option value or flag bit at all. For functions that operate on some object given in a parameter, such as a file descriptor or a port, it might instead mean that only that specific object (file descriptor, port, etc.) is unable to support the other parameters given; different file descriptors might support different ranges of parameter values.

If the entire function is not available at all in the implementation, it returns ENOSYS instead.

— Macro: int EILSEQ

While decoding a multibyte character the function came along an invalid or an incomplete sequence of bytes or the given wide character is invalid.

— Macro: int EBACKGROUND

In the GNU system, servers supporting the term protocol return this error for certain operations when the caller is not in the foreground process group of the terminal. Users do not usually see this error because functions such as read and write translate it into a SIGTTIN or SIGTTOU signal. See Job Control, for information on process groups and these signals.

— Macro: int EDIED

In the GNU system, opening a file returns this error when the file is translated by a program and the translator program dies while starting up, before it has connected to the file.

— Macro: int ED

The experienced user will know what is wrong.

— Macro: int EGREGIOUS

You did what?

— Macro: int EIEIO

Go home and have a glass of warm, dairy-fresh milk.

— Macro: int EGRATUITOUS

This error code has no purpose.

— Macro: int EBADMSG
— Macro: int EIDRM
— Macro: int EMULTIHOP
— Macro: int ENODATA
— Macro: int ENOLINK
— Macro: int ENOMSG
— Macro: int ENOSR
— Macro: int ENOSTR
— Macro: int EOVERFLOW
— Macro: int EPROTO
— Macro: int ETIME
— Macro: int ECANCELED

Operation canceled; an asynchronous operation was canceled before it completed. See Asynchronous I/O. When you call aio_cancel, the normal result is for the operations affected to complete with this error; see Cancel AIO Operations.

The following error codes are defined by the Linux/i386 kernel. They are not yet documented.

— Macro: int ERESTART
— Macro: int ECHRNG
— Macro: int EL2NSYNC
— Macro: int EL3HLT
— Macro: int EL3RST
— Macro: int ELNRNG
— Macro: int EUNATCH
— Macro: int ENOCSI
— Macro: int EL2HLT
— Macro: int EBADE
— Macro: int EBADR
— Macro: int EXFULL
— Macro: int ENOANO
— Macro: int EBADRQC
— Macro: int EBADSLT
— Macro: int EDEADLOCK
— Macro: int EBFONT
— Macro: int ENONET
— Macro: int ENOPKG
— Macro: int EADV
— Macro: int ESRMNT
— Macro: int ECOMM
— Macro: int EDOTDOT
— Macro: int ENOTUNIQ
— Macro: int EBADFD
— Macro: int EREMCHG
— Macro: int ELIBACC
— Macro: int ELIBBAD
— Macro: int ELIBSCN
— Macro: int ELIBMAX
— Macro: int ELIBEXEC
— Macro: int ESTRPIPE
— Macro: int EUCLEAN
— Macro: int ENOTNAM
— Macro: int ENAVAIL
— Macro: int EISNAM
— Macro: int EREMOTEIO
— Macro: int ENOMEDIUM
— Macro: int EMEDIUMTYPE
— Macro: int ENOKEY
— Macro: int EKEYEXPIRED
— Macro: int EKEYREVOKED
— Macro: int EKEYREJECTED
— Macro: int EOWNERDEAD
— Macro: int ENOTRECOVERABLE

Previous: Error Codes, Up: Error Reporting

2.3 Error Messages

The library has functions and variables designed to make it easy for your program to report informative error messages in the customary format about the failure of a library call. The functions strerror and perror give you the standard error message for a given error code; the variable program_invocation_short_name gives you convenient access to the name of the program that encountered the error.

— Function: char * strerror (int errnum)

The strerror function maps the error code (see Checking for Errors) specified by the errnum argument to a descriptive error message string. The return value is a pointer to this string.

The value errnum normally comes from the variable errno.

You should not modify the string returned by strerror. Also, if you make subsequent calls to strerror, the string might be overwritten. (But it's guaranteed that no library function ever calls strerror behind your back.)

The function strerror is declared in string.h.

— Function: char * strerror_r (int errnum, char *buf, size_t n)

The strerror_r function works like strerror but instead of returning the error message in a statically allocated buffer shared by all threads in the process, it returns a private copy for the thread. This might be either some permanent global data or a message string in the user supplied buffer starting at buf with the length of n bytes.

At most n characters are written (including the NUL byte) so it is up to the user to select the buffer large enough.

This function should always be used in multi-threaded programs since there is no way to guarantee the string returned by strerror really belongs to the last call of the current thread.

This function strerror_r is a GNU extension and it is declared in string.h.

— Function: void perror (const char *message)

This function prints an error message to the stream stderr; see Standard Streams. The orientation of stderr is not changed.

If you call perror with a message that is either a null pointer or an empty string, perror just prints the error message corresponding to errno, adding a trailing newline.

If you supply a non-null message argument, then perror prefixes its output with this string. It adds a colon and a space character to separate the message from the error string corresponding to errno.

The function perror is declared in stdio.h.

strerror and perror produce the exact same message for any given error code; the precise text varies from system to system. On the GNU system, the messages are fairly short; there are no multi-line messages or embedded newlines. Each error message begins with a capital letter and does not include any terminating punctuation.

Compatibility Note: The strerror function was introduced in ISO C89. Many older C systems do not support this function yet.

Many programs that don't read input from the terminal are designed to exit if any system call fails. By convention, the error message from such a program should start with the program's name, sans directories. You can find that name in the variable program_invocation_short_name; the full file name is stored the variable program_invocation_name.

— Variable: char * program_invocation_name

This variable's value is the name that was used to invoke the program running in the current process. It is the same as argv[0]. Note that this is not necessarily a useful file name; often it contains no directory names. See Program Arguments.

— Variable: char * program_invocation_short_name

This variable's value is the name that was used to invoke the program running in the current process, with directory names removed. (That is to say, it is the same as program_invocation_name minus everything up to the last slash, if any.)

The library initialization code sets up both of these variables before calling main.

Portability Note: These two variables are GNU extensions. If you want your program to work with non-GNU libraries, you must save the value of argv[0] in main, and then strip off the directory names yourself. We added these extensions to make it possible to write self-contained error-reporting subroutines that require no explicit cooperation from main.

Here is an example showing how to handle failure to open a file correctly. The function open_sesame tries to open the named file for reading and returns a stream if successful. The fopen library function returns a null pointer if it couldn't open the file for some reason. In that situation, open_sesame constructs an appropriate error message using the strerror function, and terminates the program. If we were going to make some other library calls before passing the error code to strerror, we'd have to save it in a local variable instead, because those other library functions might overwrite errno in the meantime. 

     #include <errno.h>
     #include <stdio.h>
     #include <stdlib.h>
     #include <string.h>
     
     FILE *
     open_sesame (char *name)
     {
       FILE *stream;
     
       errno = 0;
       stream = fopen (name, "r");
       if (stream == NULL)
         {
           fprintf (stderr, "%s: Couldn't open file %s; %s\n",
                    program_invocation_short_name, name, strerror (errno));
           exit (EXIT_FAILURE);
         }
       else
         return stream;
     }

Using perror has the advantage that the function is portable and available on all systems implementing ISO C. But often the text perror generates is not what is wanted and there is no way to extend or change what perror does. The GNU coding standard, for instance, requires error messages to be preceded by the program name and programs which read some input files should should provide information about the input file name and the line number in case an error is encountered while reading the file. For these occasions there are two functions available which are widely used throughout the GNU project. These functions are declared in error.h.

— Function: void error (int status, int errnum, const char *format, ...)

The error function can be used to report general problems during program execution. The format argument is a format string just like those given to the printf family of functions. The arguments required for the format can follow the format parameter. Just like perror, error also can report an error code in textual form. But unlike perror the error value is explicitly passed to the function in the errnum parameter. This eliminates the problem mentioned above that the error reporting function must be called immediately after the function causing the error since otherwise errno might have a different value.

The error prints first the program name. If the application defined a global variable error_print_progname and points it to a function this function will be called to print the program name. Otherwise the string from the global variable program_name is used. The program name is followed by a colon and a space which in turn is followed by the output produced by the format string. If the errnum parameter is non-zero the format string output is followed by a colon and a space, followed by the error message for the error code errnum. In any case is the output terminated with a newline.

The output is directed to the stderr stream. If the stderr wasn't oriented before the call it will be narrow-oriented afterwards.

The function will return unless the status parameter has a non-zero value. In this case the function will call exit with the status value for its parameter and therefore never return. If error returns the global variable error_message_count is incremented by one to keep track of the number of errors reported.

— Function: void error_at_line (int status, int errnum, const char *fname, unsigned int lineno, const char *format, ...)

The error_at_line function is very similar to the error function. The only difference are the additional parameters fname and lineno. The handling of the other parameters is identical to that of error except that between the program name and the string generated by the format string additional text is inserted.

Directly following the program name a colon, followed by the file name pointer to by fname, another colon, and a value of lineno is printed.

This additional output of course is meant to be used to locate an error in an input file (like a programming language source code file etc).

If the global variable error_one_per_line is set to a non-zero value error_at_line will avoid printing consecutive messages for the same file and line. Repetition which are not directly following each other are not caught.

Just like error this function only returned if status is zero. Otherwise exit is called with the non-zero value. If error returns the global variable error_message_count is incremented by one to keep track of the number of errors reported.

As mentioned above the error and error_at_line functions can be customized by defining a variable named error_print_progname.

— Variable: void (*) error_print_progname (void)

If the error_print_progname variable is defined to a non-zero value the function pointed to is called by error or error_at_line. It is expected to print the program name or do something similarly useful.

The function is expected to be print to the stderr stream and must be able to handle whatever orientation the stream has.

The variable is global and shared by all threads.

— Variable: unsigned int error_message_count

The error_message_count variable is incremented whenever one of the functions error or error_at_line returns. The variable is global and shared by all threads.

— Variable: int error_one_per_line

The error_one_per_line variable influences only error_at_line. Normally the error_at_line function creates output for every invocation. If error_one_per_line is set to a non-zero value error_at_line keeps track of the last file name and line number for which an error was reported and avoid directly following messages for the same file and line. This variable is global and shared by all threads.

A program which read some input file and reports errors in it could look like this: 

     {
       char *line = NULL;
       size_t len = 0;
       unsigned int lineno = 0;
     
       error_message_count = 0;
       while (! feof_unlocked (fp))
         {
           ssize_t n = getline (&line, &len, fp);
           if (n <= 0)
             /* End of file or error.  */
             break;
           ++lineno;
     
           /* Process the line.  */
           ...
     
           if (Detect error in line)
             error_at_line (0, errval, filename, lineno,
                            "some error text %s", some_variable);
         }
     
       if (error_message_count != 0)
         error (EXIT_FAILURE, 0, "%u errors found", error_message_count);
     }

error and error_at_line are clearly the functions of choice and enable the programmer to write applications which follow the GNU coding standard. The GNU libc additionally contains functions which are used in BSD for the same purpose. These functions are declared in err.h. It is generally advised to not use these functions. They are included only for compatibility.

— Function: void warn (const char *format, ...)

The warn function is roughly equivalent to a call like 

            error (0, errno, format, the parameters)

except that the global variables error respects and modifies are not used.

— Function: void vwarn (const char *format, va_list)

The vwarn function is just like warn except that the parameters for the handling of the format string format are passed in as an value of type va_list.

— Function: void warnx (const char *format, ...)

The warnx function is roughly equivalent to a call like 

            error (0, 0, format, the parameters)

except that the global variables error respects and modifies are not used. The difference to warn is that no error number string is printed.

— Function: void vwarnx (const char *format, va_list)

The vwarnx function is just like warnx except that the parameters for the handling of the format string format are passed in as an value of type va_list.

— Function: void err (int status, const char *format, ...)

The err function is roughly equivalent to a call like 

            error (status, errno, format, the parameters)

except that the global variables error respects and modifies are not used and that the program is exited even if status is zero.

— Function: void verr (int status, const char *format, va_list)

The verr function is just like err except that the parameters for the handling of the format string format are passed in as an value of type va_list.

— Function: void errx (int status, const char *format, ...)

The errx function is roughly equivalent to a call like 

            error (status, 0, format, the parameters)

except that the global variables error respects and modifies are not used and that the program is exited even if status is zero. The difference to err is that no error number string is printed.

— Function: void verrx (int status, const char *format, va_list)

The verrx function is just like errx except that the parameters for the handling of the format string format are passed in as an value of type va_list.

Next: , Previous: Error Reporting, Up: Top

3 Virtual Memory Allocation And Paging

This chapter describes how processes manage and use memory in a system that uses the GNU C library.

The GNU C Library has several functions for dynamically allocating virtual memory in various ways. They vary in generality and in efficiency. The library also provides functions for controlling paging and allocation of real memory.

Memory mapped I/O is not discussed in this chapter. See Memory-mapped I/O.

Next: , Up: Memory

3.1 Process Memory Concepts

One of the most basic resources a process has available to it is memory. There are a lot of different ways systems organize memory, but in a typical one, each process has one linear virtual address space, with addresses running from zero to some huge maximum. It need not be contiguous; i.e., not all of these addresses actually can be used to store data.

The virtual memory is divided into pages (4 kilobytes is typical). Backing each page of virtual memory is a page of real memory (called a frame) or some secondary storage, usually disk space. The disk space might be swap space or just some ordinary disk file. Actually, a page of all zeroes sometimes has nothing at all backing it – there's just a flag saying it is all zeroes. The same frame of real memory or backing store can back multiple virtual pages belonging to multiple processes. This is normally the case, for example, with virtual memory occupied by GNU C library code. The same real memory frame containing the printf function backs a virtual memory page in each of the existing processes that has a printf call in its program.

In order for a program to access any part of a virtual page, the page must at that moment be backed by (“connected to”) a real frame. But because there is usually a lot more virtual memory than real memory, the pages must move back and forth between real memory and backing store regularly, coming into real memory when a process needs to access them and then retreating to backing store when not needed anymore. This movement is called paging.

When a program attempts to access a page which is not at that moment backed by real memory, this is known as a page fault. When a page fault occurs, the kernel suspends the process, places the page into a real page frame (this is called “paging in” or “faulting in”), then resumes the process so that from the process' point of view, the page was in real memory all along. In fact, to the process, all pages always seem to be in real memory. Except for one thing: the elapsed execution time of an instruction that would normally be a few nanoseconds is suddenly much, much, longer (because the kernel normally has to do I/O to complete the page-in). For programs sensitive to that, the functions described in Locking Pages can control it. Within each virtual address space, a process has to keep track of what is at which addresses, and that process is called memory allocation. Allocation usually brings to mind meting out scarce resources, but in the case of virtual memory, that's not a major goal, because there is generally much more of it than anyone needs. Memory allocation within a process is mainly just a matter of making sure that the same byte of memory isn't used to store two different things.

Processes allocate memory in two major ways: by exec and programmatically. Actually, forking is a third way, but it's not very interesting. See Creating a Process.

Exec is the operation of creating a virtual address space for a process, loading its basic program into it, and executing the program. It is done by the “exec” family of functions (e.g. execl). The operation takes a program file (an executable), it allocates space to load all the data in the executable, loads it, and transfers control to it. That data is most notably the instructions of the program (the text), but also literals and constants in the program and even some variables: C variables with the static storage class (see Memory Allocation and C). Once that program begins to execute, it uses programmatic allocation to gain additional memory. In a C program with the GNU C library, there are two kinds of programmatic allocation: automatic and dynamic. See Memory Allocation and C.

Memory-mapped I/O is another form of dynamic virtual memory allocation. Mapping memory to a file means declaring that the contents of certain range of a process' addresses shall be identical to the contents of a specified regular file. The system makes the virtual memory initially contain the contents of the file, and if you modify the memory, the system writes the same modification to the file. Note that due to the magic of virtual memory and page faults, there is no reason for the system to do I/O to read the file, or allocate real memory for its contents, until the program accesses the virtual memory. See Memory-mapped I/O. Just as it programmatically allocates memory, the program can programmatically deallocate (free) it. You can't free the memory that was allocated by exec. When the program exits or execs, you might say that all its memory gets freed, but since in both cases the address space ceases to exist, the point is really moot. See Program Termination. A process' virtual address space is divided into segments. A segment is a contiguous range of virtual addresses. Three important segments are:

Next: , Previous: Memory Concepts, Up: Memory

3.2 Allocating Storage For Program Data

This section covers how ordinary programs manage storage for their data, including the famous malloc function and some fancier facilities special the GNU C library and GNU Compiler.

Next: , Up: Memory Allocation

3.2.1 Memory Allocation in C Programs

The C language supports two kinds of memory allocation through the variables in C programs:

A third important kind of memory allocation, dynamic allocation, is not supported by C variables but is available via GNU C library functions.

3.2.1.1 Dynamic Memory Allocation

Dynamic memory allocation is a technique in which programs determine as they are running where to store some information. You need dynamic allocation when the amount of memory you need, or how long you continue to need it, depends on factors that are not known before the program runs.

For example, you may need a block to store a line read from an input file; since there is no limit to how long a line can be, you must allocate the memory dynamically and make it dynamically larger as you read more of the line.

Or, you may need a block for each record or each definition in the input data; since you can't know in advance how many there will be, you must allocate a new block for each record or definition as you read it.

When you use dynamic allocation, the allocation of a block of memory is an action that the program requests explicitly. You call a function or macro when you want to allocate space, and specify the size with an argument. If you want to free the space, you do so by calling another function or macro. You can do these things whenever you want, as often as you want.

Dynamic allocation is not supported by C variables; there is no storage class “dynamic”, and there can never be a C variable whose value is stored in dynamically allocated space. The only way to get dynamically allocated memory is via a system call (which is generally via a GNU C library function call), and the only way to refer to dynamically allocated space is through a pointer. Because it is less convenient, and because the actual process of dynamic allocation requires more computation time, programmers generally use dynamic allocation only when neither static nor automatic allocation will serve.

For example, if you want to allocate dynamically some space to hold a struct foobar, you cannot declare a variable of type struct foobar whose contents are the dynamically allocated space. But you can declare a variable of pointer type struct foobar * and assign it the address of the space. Then you can use the operators `*' and `->' on this pointer variable to refer to the contents of the space: 

     {
       struct foobar *ptr
          = (struct foobar *) malloc (sizeof (struct foobar));
       ptr->name = x;
       ptr->next = current_foobar;
       current_foobar = ptr;
     }

Next: , Previous: Memory Allocation and C, Up: Memory Allocation

3.2.2 Unconstrained Allocation

The most general dynamic allocation facility is malloc. It allows you to allocate blocks of memory of any size at any time, make them bigger or smaller at any time, and free the blocks individually at any time (or never).

Next: , Up: Unconstrained Allocation
3.2.2.1 Basic Memory Allocation

To allocate a block of memory, call malloc. The prototype for this function is in stdlib.h.

— Function: void * malloc (size_t size)

This function returns a pointer to a newly allocated block size bytes long, or a null pointer if the block could not be allocated.

The contents of the block are undefined; you must initialize it yourself (or use calloc instead; see Allocating Cleared Space). Normally you would cast the value as a pointer to the kind of object that you want to store in the block. Here we show an example of doing so, and of initializing the space with zeros using the library function memset (see Copying and Concatenation): 

     struct foo *ptr;
     ...
     ptr = (struct foo *) malloc (sizeof (struct foo));
     if (ptr == 0) abort ();
     memset (ptr, 0, sizeof (struct foo));

You can store the result of malloc into any pointer variable without a cast, because ISO C automatically converts the type void * to another type of pointer when necessary. But the cast is necessary in contexts other than assignment operators or if you might want your code to run in traditional C.

Remember that when allocating space for a string, the argument to malloc must be one plus the length of the string. This is because a string is terminated with a null character that doesn't count in the “length” of the string but does need space. For example: 

     char *ptr;
     ...
     ptr = (char *) malloc (length + 1);

See Representation of Strings, for more information about this.

Next: , Previous: Basic Allocation, Up: Unconstrained Allocation
3.2.2.2 Examples of malloc

If no more space is available, malloc returns a null pointer. You should check the value of every call to malloc. It is useful to write a subroutine that calls malloc and reports an error if the value is a null pointer, returning only if the value is nonzero. This function is conventionally called xmalloc. Here it is: 

     void *
     xmalloc (size_t size)
     {
       register void *value = malloc (size);
       if (value == 0)
         fatal ("virtual memory exhausted");
       return value;
     }

Here is a real example of using malloc (by way of xmalloc). The function savestring will copy a sequence of characters into a newly allocated null-terminated string: 

     char *
     savestring (const char *ptr, size_t len)
     {
       register char *value = (char *) xmalloc (len + 1);
       value[len] = '\0';
       return (char *) memcpy (value, ptr, len);
     }

The block that malloc gives you is guaranteed to be aligned so that it can hold any type of data. In the GNU system, the address is always a multiple of eight on most systems, and a multiple of 16 on 64-bit systems. Only rarely is any higher boundary (such as a page boundary) necessary; for those cases, use memalign, posix_memalign or valloc (see Aligned Memory Blocks).

Note that the memory located after the end of the block is likely to be in use for something else; perhaps a block already allocated by another call to malloc. If you attempt to treat the block as longer than you asked for it to be, you are liable to destroy the data that malloc uses to keep track of its blocks, or you may destroy the contents of another block. If you have already allocated a block and discover you want it to be bigger, use realloc (see Changing Block Size).

Next: , Previous: Malloc Examples, Up: Unconstrained Allocation
3.2.2.3 Freeing Memory Allocated with malloc

When you no longer need a block that you got with malloc, use the function free to make the block available to be allocated again. The prototype for this function is in stdlib.h.

— Function: void free (void *ptr)

The free function deallocates the block of memory pointed at by ptr.

— Function: void cfree (void *ptr)

This function does the same thing as free. It's provided for backward compatibility with SunOS; you should use free instead.

Freeing a block alters the contents of the block. Do not expect to find any data (such as a pointer to the next block in a chain of blocks) in the block after freeing it. Copy whatever you need out of the block before freeing it! Here is an example of the proper way to free all the blocks in a chain, and the strings that they point to: 

     struct chain
       {
         struct chain *next;
         char *name;
       }
     
     void
     free_chain (struct chain *chain)
     {
       while (chain != 0)
         {
           struct chain *next = chain->next;
           free (chain->name);
           free (chain);
           chain = next;
         }
     }

Occasionally, free can actually return memory to the operating system and make the process smaller. Usually, all it can do is allow a later call to malloc to reuse the space. In the meantime, the space remains in your program as part of a free-list used internally by malloc.

There is no point in freeing blocks at the end of a program, because all of the program's space is given back to the system when the process terminates.

Next: , Previous: Freeing after Malloc, Up: Unconstrained Allocation
3.2.2.4 Changing the Size of a Block

Often you do not know for certain how big a block you will ultimately need at the time you must begin to use the block. For example, the block might be a buffer that you use to hold a line being read from a file; no matter how long you make the buffer initially, you may encounter a line that is longer.

You can make the block longer by calling realloc. This function is declared in stdlib.h.

— Function: void * realloc (void *ptr, size_t newsize)

The realloc function changes the size of the block whose address is ptr to be newsize.

Since the space after the end of the block may be in use, realloc may find it necessary to copy the block to a new address where more free space is available. The value of realloc is the new address of the block. If the block needs to be moved, realloc copies the old contents.

If you pass a null pointer for ptr, realloc behaves just like `malloc (newsize)'. This can be convenient, but beware that older implementations (before ISO C) may not support this behavior, and will probably crash when realloc is passed a null pointer.

Like malloc, realloc may return a null pointer if no memory space is available to make the block bigger. When this happens, the original block is untouched; it has not been modified or relocated.

In most cases it makes no difference what happens to the original block when realloc fails, because the application program cannot continue when it is out of memory, and the only thing to do is to give a fatal error message. Often it is convenient to write and use a subroutine, conventionally called xrealloc, that takes care of the error message as xmalloc does for malloc: 

     void *
     xrealloc (void *ptr, size_t size)
     {
       register void *value = realloc (ptr, size);
       if (value == 0)
         fatal ("Virtual memory exhausted");
       return value;
     }

You can also use realloc to make a block smaller. The reason you would do this is to avoid tying up a lot of memory space when only a little is needed. In several allocation implementations, making a block smaller sometimes necessitates copying it, so it can fail if no other space is available.

If the new size you specify is the same as the old size, realloc is guaranteed to change nothing and return the same address that you gave.

Next: , Previous: Changing Block Size, Up: Unconstrained Allocation
3.2.2.5 Allocating Cleared Space

The function calloc allocates memory and clears it to zero. It is declared in stdlib.h.

— Function: void * calloc (size_t count, size_t eltsize)

This function allocates a block long enough to contain a vector of count elements, each of size eltsize. Its contents are cleared to zero before calloc returns.

You could define calloc as follows: 

     void *
     calloc (size_t count, size_t eltsize)
     {
       size_t size = count * eltsize;
       void *value = malloc (size);
       if (value != 0)
         memset (value, 0, size);
       return value;
     }

But in general, it is not guaranteed that calloc calls malloc internally. Therefore, if an application provides its own malloc/realloc/free outside the C library, it should always define calloc, too.

Next: , Previous: Allocating Cleared Space, Up: Unconstrained Allocation
3.2.2.6 Efficiency Considerations for malloc

As opposed to other versions, the malloc in the GNU C Library does not round up block sizes to powers of two, neither for large nor for small sizes. Neighboring chunks can be coalesced on a free no matter what their size is. This makes the implementation suitable for all kinds of allocation patterns without generally incurring high memory waste through fragmentation.

Very large blocks (much larger than a page) are allocated with mmap (anonymous or via /dev/zero) by this implementation. This has the great advantage that these chunks are returned to the system immediately when they are freed. Therefore, it cannot happen that a large chunk becomes “locked” in between smaller ones and even after calling free wastes memory. The size threshold for mmap to be used can be adjusted with mallopt. The use of mmap can also be disabled completely.

Next: , Previous: Efficiency and Malloc, Up: Unconstrained Allocation
3.2.2.7 Allocating Aligned Memory Blocks

The address of a block returned by malloc or realloc in the GNU system is always a multiple of eight (or sixteen on 64-bit systems). If you need a block whose address is a multiple of a higher power of two than that, use memalign, posix_memalign, or valloc. memalign is declared in malloc.h and posix_memalign is declared in stdlib.h.

With the GNU library, you can use free to free the blocks that memalign, posix_memalign, and valloc return. That does not work in BSD, however—BSD does not provide any way to free such blocks.

— Function: void * memalign (size_t boundary, size_t size)

The memalign function allocates a block of size bytes whose address is a multiple of boundary. The boundary must be a power of two! The function memalign works by allocating a somewhat larger block, and then returning an address within the block that is on the specified boundary.

— Function: int posix_memalign (void **memptr, size_t alignment, size_t size)

The posix_memalign function is similar to the memalign function in that it returns a buffer of size bytes aligned to a multiple of alignment. But it adds one requirement to the parameter alignment: the value must be a power of two multiple of sizeof (void *).

If the function succeeds in allocation memory a pointer to the allocated memory is returned in *memptr and the return value is zero. Otherwise the function returns an error value indicating the problem.

This function was introduced in POSIX 1003.1d.

— Function: void * valloc (size_t size)

Using valloc is like using memalign and passing the page size as the value of the second argument. It is implemented like this: 

          void *
          valloc (size_t size)
          {
            return memalign (getpagesize (), size);
          }

Query Memory Parameters for more information about the memory subsystem.

Next: , Previous: Aligned Memory Blocks, Up: Unconstrained Allocation
3.2.2.8 Malloc Tunable Parameters

You can adjust some parameters for dynamic memory allocation with the mallopt function. This function is the general SVID/XPG interface, defined in malloc.h.

— Function: int mallopt (int param, int value)

When calling mallopt, the param argument specifies the parameter to be set, and value the new value to be set. Possible choices for param, as defined in malloc.h, are:

M_TRIM_THRESHOLD
This is the minimum size (in bytes) of the top-most, releasable chunk that will cause sbrk to be called with a negative argument in order to return memory to the system.
M_TOP_PAD
This parameter determines the amount of extra memory to obtain from the system when a call to sbrk is required. It also specifies the number of bytes to retain when shrinking the heap by calling sbrk with a negative argument. This provides the necessary hysteresis in heap size such that excessive amounts of system calls can be avoided.
M_MMAP_THRESHOLD
All chunks larger than this value are allocated outside the normal heap, using the mmap system call. This way it is guaranteed that the memory for these chunks can be returned to the system on free. Note that requests smaller than this threshold might still be allocated via mmap.
M_MMAP_MAX
The maximum number of chunks to allocate with mmap. Setting this to zero disables all use of mmap.

Next: , Previous: Malloc Tunable Parameters, Up: Unconstrained Allocation
3.2.2.9 Heap Consistency Checking

You can ask malloc to check the consistency of dynamic memory by using the mcheck function. This function is a GNU extension, declared in mcheck.h.

— Function: int mcheck (void (*abortfn) (enum mcheck_status status))

Calling mcheck tells malloc to perform occasional consistency checks. These will catch things such as writing past the end of a block that was allocated with malloc.

The abortfn argument is the function to call when an inconsistency is found. If you supply a null pointer, then mcheck uses a default function which prints a message and calls abort (see Aborting a Program). The function you supply is called with one argument, which says what sort of inconsistency was detected; its type is described below.

It is too late to begin allocation checking once you have allocated anything with malloc. So mcheck does nothing in that case. The function returns -1 if you call it too late, and 0 otherwise (when it is successful).

The easiest way to arrange to call mcheck early enough is to use the option `-lmcheck' when you link your program; then you don't need to modify your program source at all. Alternatively you might use a debugger to insert a call to mcheck whenever the program is started, for example these gdb commands will automatically call mcheck whenever the program starts: 

          (gdb) break main
          Breakpoint 1, main (argc=2, argv=0xbffff964) at whatever.c:10
          (gdb) command 1
          Type commands for when breakpoint 1 is hit, one per line.
          End with a line saying just "end".
          >call mcheck(0)
          >continue
          >end
          (gdb) ...

This will however only work if no initialization function of any object involved calls any of the malloc functions since mcheck must be called before the first such function.

— Function: enum mcheck_status mprobe (void *pointer)

The mprobe function lets you explicitly check for inconsistencies in a particular allocated block. You must have already called mcheck at the beginning of the program, to do its occasional checks; calling mprobe requests an additional consistency check to be done at the time of the call.

The argument pointer must be a pointer returned by malloc or realloc. mprobe returns a value that says what inconsistency, if any, was found. The values are described below.

— Data Type: enum mcheck_status

This enumerated type describes what kind of inconsistency was detected in an allocated block, if any. Here are the possible values:

MCHECK_DISABLED
mcheck was not called before the first allocation. No consistency checking can be done.
MCHECK_OK
No inconsistency detected.
MCHECK_HEAD
The data immediately before the block was modified. This commonly happens when an array index or pointer is decremented too far.
MCHECK_TAIL
The data immediately after the block was modified. This commonly happens when an array index or pointer is incremented too far.
MCHECK_FREE
The block was already freed.

Another possibility to check for and guard against bugs in the use of malloc, realloc and free is to set the environment variable MALLOC_CHECK_. When MALLOC_CHECK_ is set, a special (less efficient) implementation is used which is designed to be tolerant against simple errors, such as double calls of free with the same argument, or overruns of a single byte (off-by-one bugs). Not all such errors can be protected against, however, and memory leaks can result. If MALLOC_CHECK_ is set to 0, any detected heap corruption is silently ignored; if set to 1, a diagnostic is printed on stderr; if set to 2, abort is called immediately. This can be useful because otherwise a crash may happen much later, and the true cause for the problem is then very hard to track down.

There is one problem with MALLOC_CHECK_: in SUID or SGID binaries it could possibly be exploited since diverging from the normal programs behavior it now writes something to the standard error descriptor. Therefore the use of MALLOC_CHECK_ is disabled by default for SUID and SGID binaries. It can be enabled again by the system administrator by adding a file /etc/suid-debug (the content is not important it could be empty).

So, what's the difference between using MALLOC_CHECK_ and linking with `-lmcheck'? MALLOC_CHECK_ is orthogonal with respect to `-lmcheck'. `-lmcheck' has been added for backward compatibility. Both MALLOC_CHECK_ and `-lmcheck' should uncover the same bugs - but using MALLOC_CHECK_ you don't need to recompile your application.

Next: , Previous: Heap Consistency Checking, Up: Unconstrained Allocation
3.2.2.10 Memory Allocation Hooks

The GNU C library lets you modify the behavior of malloc, realloc, and free by specifying appropriate hook functions. You can use these hooks to help you debug programs that use dynamic memory allocation, for example.

The hook variables are declared in malloc.h.

— Variable: __malloc_hook

The value of this variable is a pointer to the function that malloc uses whenever it is called. You should define this function to look like malloc; that is, like: 

          void *function (size_t size, const void *caller)

The value of caller is the return address found on the stack when the malloc function was called. This value allows you to trace the memory consumption of the program.

— Variable: __realloc_hook

The value of this variable is a pointer to function that realloc uses whenever it is called. You should define this function to look like realloc; that is, like: 

          void *function (void *ptr, size_t size, const void *caller)

The value of caller is the return address found on the stack when the realloc function was called. This value allows you to trace the memory consumption of the program.

— Variable: __free_hook

The value of this variable is a pointer to function that free uses whenever it is called. You should define this function to look like free; that is, like: 

          void function (void *ptr, const void *caller)

The value of caller is the return address found on the stack when the free function was called. This value allows you to trace the memory consumption of the program.

— Variable: __memalign_hook

The value of this variable is a pointer to function that memalign uses whenever it is called. You should define this function to look like memalign; that is, like: 

          void *function (size_t alignment, size_t size, const void *caller)

The value of caller is the return address found on the stack when the memalign function was called. This value allows you to trace the memory consumption of the program.

You must make sure that the function you install as a hook for one of these functions does not call that function recursively without restoring the old value of the hook first! Otherwise, your program will get stuck in an infinite recursion. Before calling the function recursively, one should make sure to restore all the hooks to their previous value. When coming back from the recursive call, all the hooks should be resaved since a hook might modify itself.

— Variable: __malloc_initialize_hook

The value of this variable is a pointer to a function that is called once when the malloc implementation is initialized. This is a weak variable, so it can be overridden in the application with a definition like the following: 

          void (*__malloc_initialize_hook) (void) = my_init_hook;

An issue to look out for is the time at which the malloc hook functions can be safely installed. If the hook functions call the malloc-related functions recursively, it is necessary that malloc has already properly initialized itself at the time when __malloc_hook etc. is assigned to. On the other hand, if the hook functions provide a complete malloc implementation of their own, it is vital that the hooks are assigned to before the very first malloc call has completed, because otherwise a chunk obtained from the ordinary, un-hooked malloc may later be handed to __free_hook, for example.

In both cases, the problem can be solved by setting up the hooks from within a user-defined function pointed to by __malloc_initialize_hook—then the hooks will be set up safely at the right time.

Here is an example showing how to use __malloc_hook and __free_hook properly. It installs a function that prints out information every time malloc or free is called. We just assume here that realloc and memalign are not used in our program. 

     /* Prototypes for __malloc_hook, __free_hook */
     #include <malloc.h>
     
     /* Prototypes for our hooks.  */
     static void my_init_hook (void);
     static void *my_malloc_hook (size_t, const void *);
     static void my_free_hook (void*, const void *);
     
     /* Override initializing hook from the C library. */
     void (*__malloc_initialize_hook) (void) = my_init_hook;
     
     static void
     my_init_hook (void)
     {
       old_malloc_hook = __malloc_hook;
       old_free_hook = __free_hook;
       __malloc_hook = my_malloc_hook;
       __free_hook = my_free_hook;
     }
     
     static void *
     my_malloc_hook (size_t size, const void *caller)
     {
       void *result;
       /* Restore all old hooks */
       __malloc_hook = old_malloc_hook;
       __free_hook = old_free_hook;
       /* Call recursively */
       result = malloc (size);
       /* Save underlying hooks */
       old_malloc_hook = __malloc_hook;
       old_free_hook = __free_hook;
       /* printf might call malloc, so protect it too. */
       printf ("malloc (%u) returns %p\n", (unsigned int) size, result);
       /* Restore our own hooks */
       __malloc_hook = my_malloc_hook;
       __free_hook = my_free_hook;
       return result;
     }
     
     static void
     my_free_hook (void *ptr, const void *caller)
     {
       /* Restore all old hooks */
       __malloc_hook = old_malloc_hook;
       __free_hook = old_free_hook;
       /* Call recursively */
       free (ptr);
       /* Save underlying hooks */
       old_malloc_hook = __malloc_hook;
       old_free_hook = __free_hook;
       /* printf might call free, so protect it too. */
       printf ("freed pointer %p\n", ptr);
       /* Restore our own hooks */
       __malloc_hook = my_malloc_hook;
       __free_hook = my_free_hook;
     }
     
     main ()
     {
       ...
     }

The mcheck function (see Heap Consistency Checking) works by installing such hooks.

Next: , Previous: Hooks for Malloc, Up: Unconstrained Allocation
3.2.2.11 Statistics for Memory Allocation with malloc

You can get information about dynamic memory allocation by calling the mallinfo function. This function and its associated data type are declared in malloc.h; they are an extension of the standard SVID/XPG version.

— Data Type: struct mallinfo

This structure type is used to return information about the dynamic memory allocator. It contains the following members:

int arena
This is the total size of memory allocated with sbrk by malloc, in bytes.
int ordblks
This is the number of chunks not in use. (The memory allocator internally gets chunks of memory from the operating system, and then carves them up to satisfy individual malloc requests; see Efficiency and Malloc.)
int smblks
This field is unused.
int hblks
This is the total number of chunks allocated with mmap.
int hblkhd
This is the total size of memory allocated with mmap, in bytes.
int usmblks
This field is unused.
int fsmblks
This field is unused.
int uordblks
This is the total size of memory occupied by chunks handed out by malloc.
int fordblks
This is the total size of memory occupied by free (not in use) chunks.
int keepcost
This is the size of the top-most releasable chunk that normally borders the end of the heap (i.e., the high end of the virtual address space's data segment).

— Function: struct mallinfo mallinfo (void)

This function returns information about the current dynamic memory usage in a structure of type struct mallinfo.

Previous: Statistics of Malloc, Up: Unconstrained Allocation
3.2.2.12 Summary of malloc-Related Functions

Here is a summary of the functions that work with malloc:

void *malloc (size_t size)
Allocate a block of size bytes. See Basic Allocation.
void free (void *addr)
Free a block previously allocated by malloc. See Freeing after Malloc.
void *realloc (void *addr, size_t size)
Make a block previously allocated by malloc larger or smaller, possibly by copying it to a new location. See Changing Block Size.
void *calloc (size_t count, size_t eltsize)
Allocate a block of count * eltsize bytes using malloc, and set its contents to zero. See Allocating Cleared Space.
void *valloc (size_t size)
Allocate a block of size bytes, starting on a page boundary. See Aligned Memory Blocks.
void *memalign (size_t size, size_t boundary)
Allocate a block of size bytes, starting on an address that is a multiple of boundary. See Aligned Memory Blocks.
int mallopt (int param, int value)
Adjust a tunable parameter. See Malloc Tunable Parameters.
int mcheck (void (*abortfn) (void))
Tell malloc to perform occasional consistency checks on dynamically allocated memory, and to call abortfn when an inconsistency is found. See Heap Consistency Checking.
void *(*__malloc_hook) (size_t size, const void *caller)
A pointer to a function that malloc uses whenever it is called.
void *(*__realloc_hook) (void *ptr, size_t size, const void *caller)
A pointer to a function that realloc uses whenever it is called.
void (*__free_hook) (void *ptr, const void *caller)
A pointer to a function that free uses whenever it is called.
void (*__memalign_hook) (size_t size, size_t alignment, const void *caller)
A pointer to a function that memalign uses whenever it is called.
struct mallinfo mallinfo (void)
Return information about the current dynamic memory usage. See Statistics of Malloc.

Next: , Previous: Unconstrained Allocation, Up: Memory Allocation

3.2.3 Allocation Debugging

A complicated task when programming with languages which do not use garbage collected dynamic memory allocation is to find memory leaks. Long running programs must assure that dynamically allocated objects are freed at the end of their lifetime. If this does not happen the system runs out of memory, sooner or later.

The malloc implementation in the GNU C library provides some simple means to detect such leaks and obtain some information to find the location. To do this the application must be started in a special mode which is enabled by an environment variable. There are no speed penalties for the program if the debugging mode is not enabled.

Next: , Up: Allocation Debugging
3.2.3.1 How to install the tracing functionality
— Function: void mtrace (void)

When the mtrace function is called it looks for an environment variable named MALLOC_TRACE. This variable is supposed to contain a valid file name. The user must have write access. If the file already exists it is truncated. If the environment variable is not set or it does not name a valid file which can be opened for writing nothing is done. The behavior of malloc etc. is not changed. For obvious reasons this also happens if the application is installed with the SUID or SGID bit set.

If the named file is successfully opened, mtrace installs special handlers for the functions malloc, realloc, and free (see Hooks for Malloc). From then on, all uses of these functions are traced and protocolled into the file. There is now of course a speed penalty for all calls to the traced functions so tracing should not be enabled during normal use.

This function is a GNU extension and generally not available on other systems. The prototype can be found in mcheck.h.

— Function: void muntrace (void)

The muntrace function can be called after mtrace was used to enable tracing the malloc calls. If no (successful) call of mtrace was made muntrace does nothing.

Otherwise it deinstalls the handlers for malloc, realloc, and free and then closes the protocol file. No calls are protocolled anymore and the program runs again at full speed.

This function is a GNU extension and generally not available on other systems. The prototype can be found in mcheck.h.

Next: , Previous: Tracing malloc, Up: Allocation Debugging
3.2.3.2 Example program excerpts

Even though the tracing functionality does not influence the runtime behavior of the program it is not a good idea to call mtrace in all programs. Just imagine that you debug a program using mtrace and all other programs used in the debugging session also trace their malloc calls. The output file would be the same for all programs and thus is unusable. Therefore one should call mtrace only if compiled for debugging. A program could therefore start like this: 

     #include <mcheck.h>
     
     int
     main (int argc, char *argv[])
     {
     #ifdef DEBUGGING
       mtrace ();
     #endif
       ...
     }

This is all what is needed if you want to trace the calls during the whole runtime of the program. Alternatively you can stop the tracing at any time with a call to muntrace. It is even possible to restart the tracing again with a new call to mtrace. But this can cause unreliable results since there may be calls of the functions which are not called. Please note that not only the application uses the traced functions, also libraries (including the C library itself) use these functions.

This last point is also why it is no good idea to call muntrace before the program terminated. The libraries are informed about the termination of the program only after the program returns from main or calls exit and so cannot free the memory they use before this time.

So the best thing one can do is to call mtrace as the very first function in the program and never call muntrace. So the program traces almost all uses of the malloc functions (except those calls which are executed by constructors of the program or used libraries).

Next: , Previous: Using the Memory Debugger, Up: Allocation Debugging
3.2.3.3 Some more or less clever ideas

You know the situation. The program is prepared for debugging and in all debugging sessions it runs well. But once it is started without debugging the error shows up. A typical example is a memory leak that becomes visible only when we turn off the debugging. If you foresee such situations you can still win. Simply use something equivalent to the following little program: 

     #include <mcheck.h>
     #include <signal.h>
     
     static void
     enable (int sig)
     {
       mtrace ();
       signal (SIGUSR1, enable);
     }
     
     static void
     disable (int sig)
     {
       muntrace ();
       signal (SIGUSR2, disable);
     }
     
     int
     main (int argc, char *argv[])
     {
       ...
     
       signal (SIGUSR1, enable);
       signal (SIGUSR2, disable);
     
       ...
     }

I.e., the user can start the memory debugger any time s/he wants if the program was started with MALLOC_TRACE set in the environment. The output will of course not show the allocations which happened before the first signal but if there is a memory leak this will show up nevertheless.

Previous: Tips for the Memory Debugger, Up: Allocation Debugging
3.2.3.4 Interpreting the traces

If you take a look at the output it will look similar to this: 

     = Start
      [0x8048209] - 0x8064cc8
      [0x8048209] - 0x8064ce0
      [0x8048209] - 0x8064cf8
      [0x80481eb] + 0x8064c48 0x14
      [0x80481eb] + 0x8064c60 0x14
      [0x80481eb] + 0x8064c78 0x14
      [0x80481eb] + 0x8064c90 0x14
     = End

What this all means is not really important since the trace file is not meant to be read by a human. Therefore no attention is given to readability. Instead there is a program which comes with the GNU C library which interprets the traces and outputs a summary in an user-friendly way. The program is called mtrace (it is in fact a Perl script) and it takes one or two arguments. In any case the name of the file with the trace output must be specified. If an optional argument precedes the name of the trace file this must be the name of the program which generated the trace. 

     drepper$ mtrace tst-mtrace log
     No memory leaks.

In this case the program tst-mtrace was run and it produced a trace file log. The message printed by mtrace shows there are no problems with the code, all allocated memory was freed afterwards.

If we call mtrace on the example trace given above we would get a different outout: 

     drepper$ mtrace errlog
     - 0x08064cc8 Free 2 was never alloc'd 0x8048209
     - 0x08064ce0 Free 3 was never alloc'd 0x8048209
     - 0x08064cf8 Free 4 was never alloc'd 0x8048209
     
     Memory not freed:
     -----------------
        Address     Size     Caller
     0x08064c48     0x14  at 0x80481eb
     0x08064c60     0x14  at 0x80481eb
     0x08064c78     0x14  at 0x80481eb
     0x08064c90     0x14  at 0x80481eb

We have called mtrace with only one argument and so the script has no chance to find out what is meant with the addresses given in the trace. We can do better: 

     drepper$ mtrace tst errlog
     - 0x08064cc8 Free 2 was never alloc'd /home/drepper/tst.c:39
     - 0x08064ce0 Free 3 was never alloc'd /home/drepper/tst.c:39
     - 0x08064cf8 Free 4 was never alloc'd /home/drepper/tst.c:39
     
     Memory not freed:
     -----------------
        Address     Size     Caller
     0x08064c48     0x14  at /home/drepper/tst.c:33
     0x08064c60     0x14  at /home/drepper/tst.c:33
     0x08064c78     0x14  at /home/drepper/tst.c:33
     0x08064c90     0x14  at /home/drepper/tst.c:33

Suddenly the output makes much more sense and the user can see immediately where the function calls causing the trouble can be found.

Interpreting this output is not complicated. There are at most two different situations being detected. First, free was called for pointers which were never returned by one of the allocation functions. This is usually a very bad problem and what this looks like is shown in the first three lines of the output. Situations like this are quite rare and if they appear they show up very drastically: the program normally crashes.

The other situation which is much harder to detect are memory leaks. As you can see in the output the mtrace function collects all this information and so can say that the program calls an allocation function from line 33 in the source file /home/drepper/tst-mtrace.c four times without freeing this memory before the program terminates. Whether this is a real problem remains to be investigated.

Next: , Previous: Allocation Debugging, Up: Memory Allocation

3.2.4 Obstacks

An obstack is a pool of memory containing a stack of objects. You can create any number of separate obstacks, and then allocate objects in specified obstacks. Within each obstack, the last object allocated must always be the first one freed, but distinct obstacks are independent of each other.

Aside from this one constraint of order of freeing, obstacks are totally general: an obstack can contain any number of objects of any size. They are implemented with macros, so allocation is usually very fast as long as the objects are usually small. And the only space overhead per object is the padding needed to start each object on a suitable boundary.

Next: , Up: Obstacks
3.2.4.1 Creating Obstacks

The utilities for manipulating obstacks are declared in the header file obstack.h.

— Data Type: struct obstack

An obstack is represented by a data structure of type struct obstack. This structure has a small fixed size; it records the status of the obstack and how to find the space in which objects are allocated. It does not contain any of the objects themselves. You should not try to access the contents of the structure directly; use only the functions described in this chapter.

You can declare variables of type struct obstack and use them as obstacks, or you can allocate obstacks dynamically like any other kind of object. Dynamic allocation of obstacks allows your program to have a variable number of different stacks. (You can even allocate an obstack structure in another obstack, but this is rarely useful.)

All the functions that work with obstacks require you to specify which obstack to use. You do this with a pointer of type struct obstack *. In the following, we often say “an obstack” when strictly speaking the object at hand is such a pointer.

The objects in the obstack are packed into large blocks called chunks. The struct obstack structure points to a chain of the chunks currently in use.

The obstack library obtains a new chunk whenever you allocate an object that won't fit in the previous chunk. Since the obstack library manages chunks automatically, you don't need to pay much attention to them, but you do need to supply a function which the obstack library should use to get a chunk. Usually you supply a function which uses malloc directly or indirectly. You must also supply a function to free a chunk. These matters are described in the following section.

Next: , Previous: Creating Obstacks, Up: Obstacks
3.2.4.2 Preparing for Using Obstacks

Each source file in which you plan to use the obstack functions must include the header file obstack.h, like this: 

     #include <obstack.h>

Also, if the source file uses the macro obstack_init, it must declare or define two functions or macros that will be called by the obstack library. One, obstack_chunk_alloc, is used to allocate the chunks of memory into which objects are packed. The other, obstack_chunk_free, is used to return chunks when the objects in them are freed. These macros should appear before any use of obstacks in the source file.

Usually these are defined to use malloc via the intermediary xmalloc (see Unconstrained Allocation). This is done with the following pair of macro definitions: 

     #define obstack_chunk_alloc xmalloc
     #define obstack_chunk_free free

Though the memory you get using obstacks really comes from malloc, using obstacks is faster because malloc is called less often, for larger blocks of memory. See Obstack Chunks, for full details.

At run time, before the program can use a struct obstack object as an obstack, it must initialize the obstack by calling obstack_init.

— Function: int obstack_init (struct obstack *obstack-ptr)

Initialize obstack obstack-ptr for allocation of objects. This function calls the obstack's obstack_chunk_alloc function. If allocation of memory fails, the function pointed to by obstack_alloc_failed_handler is called. The obstack_init function always returns 1 (Compatibility notice: Former versions of obstack returned 0 if allocation failed).

Here are two examples of how to allocate the space for an obstack and initialize it. First, an obstack that is a static variable: 

     static struct obstack myobstack;
     ...
     obstack_init (&myobstack);

Second, an obstack that is itself dynamically allocated: 

     struct obstack *myobstack_ptr
       = (struct obstack *) xmalloc (sizeof (struct obstack));
     
     obstack_init (myobstack_ptr);
— Variable: obstack_alloc_failed_handler

The value of this variable is a pointer to a function that obstack uses when obstack_chunk_alloc fails to allocate memory. The default action is to print a message and abort. You should supply a function that either calls exit (see Program Termination) or longjmp (see Non-Local Exits) and doesn't return. 

          void my_obstack_alloc_failed (void)
          ...
          obstack_alloc_failed_handler = &my_obstack_alloc_failed;

Next: , Previous: Preparing for Obstacks, Up: Obstacks
3.2.4.3 Allocation in an Obstack

The most direct way to allocate an object in an obstack is with obstack_alloc, which is invoked almost like malloc.

— Function: void * obstack_alloc (struct obstack *obstack-ptr, int size)

This allocates an uninitialized block of size bytes in an obstack and returns its address. Here obstack-ptr specifies which obstack to allocate the block in; it is the address of the struct obstack object which represents the obstack. Each obstack function or macro requires you to specify an obstack-ptr as the first argument.

This function calls the obstack's obstack_chunk_alloc function if it needs to allocate a new chunk of memory; it calls obstack_alloc_failed_handler if allocation of memory by obstack_chunk_alloc failed.

For example, here is a function that allocates a copy of a string str in a specific obstack, which is in the variable string_obstack: 

     struct obstack string_obstack;
     
     char *
     copystring (char *string)
     {
       size_t len = strlen (string) + 1;
       char *s = (char *) obstack_alloc (&string_obstack, len);
       memcpy (s, string, len);
       return s;
     }

To allocate a block with specified contents, use the function obstack_copy, declared like this:

— Function: void * obstack_copy (struct obstack *obstack-ptr, void *address, int size)

This allocates a block and initializes it by copying size bytes of data starting at address. It calls obstack_alloc_failed_handler if allocation of memory by obstack_chunk_alloc failed.

— Function: void * obstack_copy0 (struct obstack *obstack-ptr, void *address, int size)

Like obstack_copy, but appends an extra byte containing a null character. This extra byte is not counted in the argument size.

The obstack_copy0 function is convenient for copying a sequence of characters into an obstack as a null-terminated string. Here is an example of its use: 

     char *
     obstack_savestring (char *addr, int size)
     {
       return obstack_copy0 (&myobstack, addr, size);
     }

Contrast this with the previous example of savestring using malloc (see Basic Allocation).

Next: , Previous: Allocation in an Obstack, Up: Obstacks
3.2.4.4 Freeing Objects in an Obstack

To free an object allocated in an obstack, use the function obstack_free. Since the obstack is a stack of objects, freeing one object automatically frees all other objects allocated more recently in the same obstack.

— Function: void obstack_free (struct obstack *obstack-ptr, void *object)

If object is a null pointer, everything allocated in the obstack is freed. Otherwise, object must be the address of an object allocated in the obstack. Then object is freed, along with everything allocated in obstack since object.

Note that if object is a null pointer, the result is an uninitialized obstack. To free all memory in an obstack but leave it valid for further allocation, call obstack_free with the address of the first object allocated on the obstack: 

     obstack_free (obstack_ptr, first_object_allocated_ptr);

Recall that the objects in an obstack are grouped into chunks. When all the objects in a chunk become free, the obstack library automatically frees the chunk (see Preparing for Obstacks). Then other obstacks, or non-obstack allocation, can reuse the space of the chunk.

Next: , Previous: Freeing Obstack Objects, Up: Obstacks
3.2.4.5 Obstack Functions and Macros

The interfaces for using obstacks may be defined either as functions or as macros, depending on the compiler. The obstack facility works with all C compilers, including both ISO C and traditional C, but there are precautions you must take if you plan to use compilers other than GNU C.

If you are using an old-fashioned non-ISO C compiler, all the obstack “functions” are actually defined only as macros. You can call these macros like functions, but you cannot use them in any other way (for example, you cannot take their address).

Calling the macros requires a special precaution: namely, the first operand (the obstack pointer) may not contain any side effects, because it may be computed more than once. For example, if you write this: 

     obstack_alloc (get_obstack (), 4);

you will find that get_obstack may be called several times. If you use *obstack_list_ptr++ as the obstack pointer argument, you will get very strange results since the incrementation may occur several times.

In ISO C, each function has both a macro definition and a function definition. The function definition is used if you take the address of the function without calling it. An ordinary call uses the macro definition by default, but you can request the function definition instead by writing the function name in parentheses, as shown here: 

     char *x;
     void *(*funcp) ();
     /* Use the macro.  */
     x = (char *) obstack_alloc (obptr, size);
     /* Call the function.  */
     x = (char *) (obstack_alloc) (obptr, size);
     /* Take the address of the function.  */
     funcp = obstack_alloc;

This is the same situation that exists in ISO C for the standard library functions. See Macro Definitions.

Warning: When you do use the macros, you must observe the precaution of avoiding side effects in the first operand, even in ISO C.

If you use the GNU C compiler, this precaution is not necessary, because various language extensions in GNU C permit defining the macros so as to compute each argument only once.

Next: , Previous: Obstack Functions, Up: Obstacks
3.2.4.6 Growing Objects

Because memory in obstack chunks is used sequentially, it is possible to build up an object step by step, adding one or more bytes at a time to the end of the object. With this technique, you do not need to know how much data you will put in the object until you come to the end of it. We call this the technique of growing objects. The special functions for adding data to the growing object are described in this section.

You don't need to do anything special when you start to grow an object. Using one of the functions to add data to the object automatically starts it. However, it is necessary to say explicitly when the object is finished. This is done with the function obstack_finish.

The actual address of the object thus built up is not known until the object is finished. Until then, it always remains possible that you will add so much data that the object must be copied into a new chunk.

While the obstack is in use for a growing object, you cannot use it for ordinary allocation of another object. If you try to do so, the space already added to the growing object will become part of the other object.

— Function: void obstack_blank (struct obstack *obstack-ptr, int size)

The most basic function for adding to a growing object is obstack_blank, which adds space without initializing it.

— Function: void obstack_grow (struct obstack *obstack-ptr, void *data, int size)

To add a block of initialized space, use obstack_grow, which is the growing-object analogue of obstack_copy. It adds size bytes of data to the growing object, copying the contents from data.

— Function: void obstack_grow0 (struct obstack *obstack-ptr, void *data, int size)

This is the growing-object analogue of obstack_copy0. It adds size bytes copied from data, followed by an additional null character.

— Function: void obstack_1grow (struct obstack *obstack-ptr, char c)

To add one character at a time, use the function obstack_1grow. It adds a single byte containing c to the growing object.

— Function: void obstack_ptr_grow (struct obstack *obstack-ptr, void *data)

Adding the value of a pointer one can use the function obstack_ptr_grow. It adds sizeof (void *) bytes containing the value of data.

— Function: void obstack_int_grow (struct obstack *obstack-ptr, int data)

A single value of type int can be added by using the obstack_int_grow function. It adds sizeof (int) bytes to the growing object and initializes them with the value of data.

— Function: void * obstack_finish (struct obstack *obstack-ptr)

When you are finished growing the object, use the function obstack_finish to close it off and return its final address.

Once you have finished the object, the obstack is available for ordinary allocation or for growing another object.

This function can return a null pointer under the same conditions as obstack_alloc (see Allocation in an Obstack).

When you build an object by growing it, you will probably need to know afterward how long it became. You need not keep track of this as you grow the object, because you can find out the length from the obstack just before finishing the object with the function obstack_object_size, declared as follows:

— Function: int obstack_object_size (struct obstack *obstack-ptr)

This function returns the current size of the growing object, in bytes. Remember to call this function before finishing the object. After it is finished, obstack_object_size will return zero.

If you have started growing an object and wish to cancel it, you should finish it and then free it, like this: 

     obstack_free (obstack_ptr, obstack_finish (obstack_ptr));

This has no effect if no object was growing.

You can use obstack_blank with a negative size argument to make the current object smaller. Just don't try to shrink it beyond zero length—there's no telling what will happen if you do that.

Next: , Previous: Growing Objects, Up: Obstacks
3.2.4.7 Extra Fast Growing Objects

The usual functions for growing objects incur overhead for checking whether there is room for the new growth in the current chunk. If you are frequently constructing objects in small steps of growth, this overhead can be significant.

You can reduce the overhead by using special “fast growth” functions that grow the object without checking. In order to have a robust program, you must do the checking yourself. If you do this checking in the simplest way each time you are about to add data to the object, you have not saved anything, because that is what the ordinary growth functions do. But if you can arrange to check less often, or check more efficiently, then you make the program faster.

The function obstack_room returns the amount of room available in the current chunk. It is declared as follows:

— Function: int obstack_room (struct obstack *obstack-ptr)

This returns the number of bytes that can be added safely to the current growing object (or to an object about to be started) in obstack obstack using the fast growth functions.

While you know there is room, you can use these fast growth functions for adding data to a growing object:

— Function: void obstack_1grow_fast (struct obstack *obstack-ptr, char c)

The function obstack_1grow_fast adds one byte containing the character c to the growing object in obstack obstack-ptr.

— Function: void obstack_ptr_grow_fast (struct obstack *obstack-ptr, void *data)

The function obstack_ptr_grow_fast adds sizeof (void *) bytes containing the value of data to the growing object in obstack obstack-ptr.

— Function: void obstack_int_grow_fast (struct obstack *obstack-ptr, int data)

The function obstack_int_grow_fast adds sizeof (int) bytes containing the value of data to the growing object in obstack obstack-ptr.

— Function: void obstack_blank_fast (struct obstack *obstack-ptr, int size)

The function obstack_blank_fast adds size bytes to the growing object in obstack obstack-ptr without initializing them.

When you check for space using obstack_room and there is not enough room for what you want to add, the fast growth functions are not safe. In this case, simply use the corresponding ordinary growth function instead. Very soon this will copy the object to a new chunk; then there will be lots of room available again.

So, each time you use an ordinary growth function, check afterward for sufficient space using obstack_room. Once the object is copied to a new chunk, there will be plenty of space again, so the program will start using the fast growth functions again.

Here is an example: 

     void
     add_string (struct obstack *obstack, const char *ptr, int len)
     {
       while (len > 0)
         {
           int room = obstack_room (obstack);
           if (room == 0)
             {
               /* Not enough room. Add one character slowly,
                  which may copy to a new chunk and make room.  */
               obstack_1grow (obstack, *ptr++);
               len--;
             }
           else
             {
               if (room > len)
                 room = len;
               /* Add fast as much as we have room for. */
               len -= room;
               while (room-- > 0)
                 obstack_1grow_fast (obstack, *ptr++);
             }
         }
     }

Next: , Previous: Extra Fast Growing, Up: Obstacks
3.2.4.8 Status of an Obstack

Here are functions that provide information on the current status of allocation in an obstack. You can use them to learn about an object while still growing it.

— Function: void * obstack_base (struct obstack *obstack-ptr)

This function returns the tentative address of the beginning of the currently growing object in obstack-ptr. If you finish the object immediately, it will have that address. If you make it larger first, it may outgrow the current chunk—then its address will change!

If no object is growing, this value says where the next object you allocate will start (once again assuming it fits in the current chunk).

— Function: void * obstack_next_free (struct obstack *obstack-ptr)

This function returns the address of the first free byte in the current chunk of obstack obstack-ptr. This is the end of the currently growing object. If no object is growing, obstack_next_free returns the same value as obstack_base.

— Function: int obstack_object_size (struct obstack *obstack-ptr)

This function returns the size in bytes of the currently growing object. This is equivalent to 

          obstack_next_free (obstack-ptr) - obstack_base (obstack-ptr)

Next: , Previous: Status of an Obstack, Up: Obstacks
3.2.4.9 Alignment of Data in Obstacks

Each obstack has an alignment boundary; each object allocated in the obstack automatically starts on an address that is a multiple of the specified boundary. By default, this boundary is aligned so that the object can hold any type of data.

To access an obstack's alignment boundary, use the macro obstack_alignment_mask, whose function prototype looks like this:

— Macro: int obstack_alignment_mask (struct obstack *obstack-ptr)

The value is a bit mask; a bit that is 1 indicates that the corresponding bit in the address of an object should be 0. The mask value should be one less than a power of 2; the effect is that all object addresses are multiples of that power of 2. The default value of the mask is a value that allows aligned objects to hold any type of data: for example, if its value is 3, any type of data can be stored at locations whose addresses are multiples of 4. A mask value of 0 means an object can start on any multiple of 1 (that is, no alignment is required).

The expansion of the macro obstack_alignment_mask is an lvalue, so you can alter the mask by assignment. For example, this statement: 

          obstack_alignment_mask (obstack_ptr) = 0;

has the effect of turning off alignment processing in the specified obstack.

Note that a change in alignment mask does not take effect until after the next time an object is allocated or finished in the obstack. If you are not growing an object, you can make the new alignment mask take effect immediately by calling obstack_finish. This will finish a zero-length object and then do proper alignment for the next object.

Next: , Previous: Obstacks Data Alignment, Up: Obstacks
3.2.4.10 Obstack Chunks

Obstacks work by allocating space for themselves in large chunks, and then parceling out space in the chunks to satisfy your requests. Chunks are normally 4096 bytes long unless you specify a different chunk size. The chunk size includes 8 bytes of overhead that are not actually used for storing objects. Regardless of the specified size, longer chunks will be allocated when necessary for long objects.

The obstack library allocates chunks by calling the function obstack_chunk_alloc, which you must define. When a chunk is no longer needed because you have freed all the objects in it, the obstack library frees the chunk by calling obstack_chunk_free, which you must also define.

These two must be defined (as macros) or declared (as functions) in each source file that uses obstack_init (see Creating Obstacks). Most often they are defined as macros like this: 

     #define obstack_chunk_alloc malloc
     #define obstack_chunk_free free

Note that these are simple macros (no arguments). Macro definitions with arguments will not work! It is necessary that obstack_chunk_alloc or obstack_chunk_free, alone, expand into a function name if it is not itself a function name.

If you allocate chunks with malloc, the chunk size should be a power of 2. The default chunk size, 4096, was chosen because it is long enough to satisfy many typical requests on the obstack yet short enough not to waste too much memory in the portion of the last chunk not yet used.

— Macro: int obstack_chunk_size (struct obstack *obstack-ptr)

This returns the chunk size of the given obstack.

Since this macro expands to an lvalue, you can specify a new chunk size by assigning it a new value. Doing so does not affect the chunks already allocated, but will change the size of chunks allocated for that particular obstack in the future. It is unlikely to be useful to make the chunk size smaller, but making it larger might improve efficiency if you are allocating many objects whose size is comparable to the chunk size. Here is how to do so cleanly: 

     if (obstack_chunk_size (obstack_ptr) < new-chunk-size)
       obstack_chunk_size (obstack_ptr) = new-chunk-size;

Previous: Obstack Chunks, Up: Obstacks
3.2.4.11 Summary of Obstack Functions

Here is a summary of all the functions associated with obstacks. Each takes the address of an obstack (struct obstack *) as its first argument.

void obstack_init (struct obstack *obstack-ptr)
Initialize use of an obstack. See Creating Obstacks.
void *obstack_alloc (struct obstack *obstack-ptr, int size)
Allocate an object of size uninitialized bytes. See Allocation in an Obstack.
void *obstack_copy (struct obstack *obstack-ptr, void *address, int size)
Allocate an object of size bytes, with contents copied from address. See Allocation in an Obstack.
void *obstack_copy0 (struct obstack *obstack-ptr, void *address, int size)
Allocate an object of size+1 bytes, with size of them copied from address, followed by a null character at the end. See Allocation in an Obstack.
void obstack_free (struct obstack *obstack-ptr, void *object)
Free object (and everything allocated in the specified obstack more recently than object). See Freeing Obstack Objects.
void obstack_blank (struct obstack *obstack-ptr, int size)
Add size uninitialized bytes to a growing object. See Growing Objects.
void obstack_grow (struct obstack *obstack-ptr, void *address, int size)
Add size bytes, copied from address, to a growing object. See Growing Objects.
void obstack_grow0 (struct obstack *obstack-ptr, void *address, int size)
Add size bytes, copied from address, to a growing object, and then add another byte containing a null character. See Growing Objects.
void obstack_1grow (struct obstack *obstack-ptr, char data-char)
Add one byte containing data-char to a growing object. See Growing Objects.
void *obstack_finish (struct obstack *obstack-ptr)
Finalize the object that is growing and return its permanent address. See Growing Objects.
int obstack_object_size (struct obstack *obstack-ptr)
Get the current size of the currently growing object. See Growing Objects.
void obstack_blank_fast (struct obstack *obstack-ptr, int size)
Add size uninitialized bytes to a growing object without checking that there is enough room. See Extra Fast Growing.
void obstack_1grow_fast (struct obstack *obstack-ptr, char data-char)
Add one byte containing data-char to a growing object without checking that there is enough room. See Extra Fast Growing.
int obstack_room (struct obstack *obstack-ptr)
Get the amount of room now available for growing the current object. See Extra Fast Growing.
int obstack_alignment_mask (struct obstack *obstack-ptr)
The mask used for aligning the beginning of an object. This is an lvalue. See Obstacks Data Alignment.
int obstack_chunk_size (struct obstack *obstack-ptr)
The size for allocating chunks. This is an lvalue. See Obstack Chunks.
void *obstack_base (struct obstack *obstack-ptr)
Tentative starting address of the currently growing object. See Status of an Obstack.
void *obstack_next_free (struct obstack *obstack-ptr)
Address just after the end of the currently growing object. See Status of an Obstack.

Previous: Obstacks, Up: Memory Allocation

3.2.5 Automatic Storage with Variable Size

The function alloca supports a kind of half-dynamic allocation in which blocks are allocated dynamically but freed automatically.

Allocating a block with alloca is an explicit action; you can allocate as many blocks as you wish, and compute the size at run time. But all the blocks are freed when you exit the function that alloca was called from, just as if they were automatic variables declared in that function. There is no way to free the space explicitly.

The prototype for alloca is in stdlib.h. This function is a BSD extension.

— Function: void * alloca (size_t size);

The return value of alloca is the address of a block of size bytes of memory, allocated in the stack frame of the calling function.

Do not use alloca inside the arguments of a function call—you will get unpredictable results, because the stack space for the alloca would appear on the stack in the middle of the space for the function arguments. An example of what to avoid is foo (x, alloca (4), y).

Next: , Up: Variable Size Automatic
3.2.5.1 alloca Example

As an example of the use of alloca, here is a function that opens a file name made from concatenating two argument strings, and returns a file descriptor or minus one signifying failure: 

     int
     open2 (char *str1, char *str2, int flags, int mode)
     {
       char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
       stpcpy (stpcpy (name, str1), str2);
       return open (name, flags, mode);
     }

Here is how you would get the same results with malloc and free: 

     int
     open2 (char *str1, char *str2, int flags, int mode)
     {
       char *name = (char *) malloc (strlen (str1) + strlen (str2) + 1);
       int desc;
       if (name == 0)
         fatal ("virtual memory exceeded");
       stpcpy (stpcpy (name, str1), str2);
       desc = open (name, flags, mode);
       free (name);
       return desc;
     }

As you can see, it is simpler with alloca. But alloca has other, more important advantages, and some disadvantages.

Next: , Previous: Alloca Example, Up: Variable Size Automatic
3.2.5.2 Advantages of alloca

Here are the reasons why alloca may be preferable to malloc:

Next: , Previous: Advantages of Alloca, Up: Variable Size Automatic
3.2.5.3 Disadvantages of alloca

These are the disadvantages of alloca in comparison with malloc:

Previous: Disadvantages of Alloca, Up: Variable Size Automatic
3.2.5.4 GNU C Variable-Size Arrays

In GNU C, you can replace most uses of alloca with an array of variable size. Here is how open2 would look then: 

     int open2 (char *str1, char *str2, int flags, int mode)
     {
       char name[strlen (str1) + strlen (str2) + 1];
       stpcpy (stpcpy (name, str1), str2);
       return open (name, flags, mode);
     }

But alloca is not always equivalent to a variable-sized array, for several reasons:

Note: If you mix use of alloca and variable-sized arrays within one function, exiting a scope in which a variable-sized array was declared frees all blocks allocated with alloca during the execution of that scope.

Previous: Locking Pages, Up: Memory

3.3 Resizing the Data Segment

The symbols in this section are declared in unistd.h.

You will not normally use the functions in this section, because the functions described in Memory Allocation are easier to use. Those are interfaces to a GNU C Library memory allocator that uses the functions below itself. The functions below are simple interfaces to system calls.

— Function: int brk (void *addr)

brk sets the high end of the calling process' data segment to addr.

The address of the end of a segment is defined to be the address of the last byte in the segment plus 1.

The function has no effect if addr is lower than the low end of the data segment. (This is considered success, by the way).

The function fails if it would cause the data segment to overlap another segment or exceed the process' data storage limit (see Limits on Resources).

The function is named for a common historical case where data storage and the stack are in the same segment. Data storage allocation grows upward from the bottom of the segment while the stack grows downward toward it from the top of the segment and the curtain between them is called the break.

The return value is zero on success. On failure, the return value is -1 and errno is set accordingly. The following errno values are specific to this function:

ENOMEM
The request would cause the data segment to overlap another segment or exceed the process' data storage limit.
— Function: void *sbrk (ptrdiff_t delta)

This function is the same as brk except that you specify the new end of the data segment as an offset delta from the current end and on success the return value is the address of the resulting end of the data segment instead of zero.

This means you can use `sbrk(0)' to find out what the current end of the data segment is.

Next: , Previous: Memory Allocation, Up: Memory

3.4 Locking Pages

You can tell the system to associate a particular virtual memory page with a real page frame and keep it that way — i.e., cause the page to be paged in if it isn't already and mark it so it will never be paged out and consequently will never cause a page fault. This is called locking a page.

The functions in this chapter lock and unlock the calling process' pages.

Next: , Up: Locking Pages

3.4.1 Why Lock Pages

Because page faults cause paged out pages to be paged in transparently, a process rarely needs to be concerned about locking pages. However, there are two reasons people sometimes are:

Be aware that when you lock a page, that's one fewer page frame that can be used to back other virtual memory (by the same or other processes), which can mean more page faults, which means the system runs more slowly. In fact, if you lock enough memory, some programs may not be able to run at all for lack of real memory.

Next: , Previous: Why Lock Pages, Up: Locking Pages

3.4.2 Locked Memory Details

A memory lock is associated with a virtual page, not a real frame. The paging rule is: If a frame backs at least one locked page, don't page it out.

Memory locks do not stack. I.e., you can't lock a particular page twice so that it has to be unlocked twice before it is truly unlocked. It is either locked or it isn't.

A memory lock persists until the process that owns the memory explicitly unlocks it. (But process termination and exec cause the virtual memory to cease to exist, which you might say means it isn't locked any more).

Memory locks are not inherited by child processes. (But note that on a modern Unix system, immediately after a fork, the parent's and the child's virtual address space are backed by the same real page frames, so the child enjoys the parent's locks). See Creating a Process.

Because of its ability to impact other processes, only the superuser can lock a page. Any process can unlock its own page.

The system sets limits on the amount of memory a process can have locked and the amount of real memory it can have dedicated to it. See Limits on Resources.

In Linux, locked pages aren't as locked as you might think. Two virtual pages that are not shared memory can nonetheless be backed by the same real frame. The kernel does this in the name of efficiency when it knows both virtual pages contain identical data, and does it even if one or both of the virtual pages are locked.

But when a process modifies one of those pages, the kernel must get it a separate frame and fill it with the page's data. This is known as a copy-on-write page fault. It takes a small amount of time and in a pathological case, getting that frame may require I/O. To make sure this doesn't happen to your program, don't just lock the pages. Write to them as well, unless you know you won't write to them ever. And to make sure you have pre-allocated frames for your stack, enter a scope that declares a C automatic variable larger than the maximum stack size you will need, set it to something, then return from its scope.

Previous: Locked Memory Details, Up: Locking Pages

3.4.3 Functions To Lock And Unlock Pages

The symbols in this section are declared in sys/mman.h. These functions are defined by POSIX.1b, but their availability depends on your kernel. If your kernel doesn't allow these functions, they exist but always fail. They are available with a Linux kernel.

Portability Note: POSIX.1b requires that when the mlock and munlock functions are available, the file unistd.h define the macro _POSIX_MEMLOCK_RANGE and the file limits.h define the macro PAGESIZE to be the size of a memory page in bytes. It requires that when the mlockall and munlockall functions are available, the unistd.h file define the macro _POSIX_MEMLOCK. The GNU C library conforms to this requirement.

— Function: int mlock (const void *addr, size_t len)

mlock locks a range of the calling process' virtual pages.

The range of memory starts at address addr and is len bytes long. Actually, since you must lock whole pages, it is the range of pages that include any part of the specified range.

When the function returns successfully, each of those pages is backed by (connected to) a real frame (is resident) and is marked to stay that way. This means the function may cause page-ins and have to wait for them.

When the function fails, it does not affect the lock status of any pages.

The return value is zero if the function succeeds. Otherwise, it is -1 and errno is set accordingly. errno values specific to this function are:

ENOMEM
  • At least some of the specified address range does not exist in the calling process' virtual address space.
  • The locking would cause the process to exceed its locked page limit.

EPERM
The calling process is not superuser.
EINVAL
len is not positive.
ENOSYS
The kernel does not provide mlock capability.

You can lock all a process' memory with mlockall. You unlock memory with munlock or munlockall.

To avoid all page faults in a C program, you have to use mlockall, because some of the memory a program uses is hidden from the C code, e.g. the stack and automatic variables, and you wouldn't know what address to tell mlock.

— Function: int munlock (const void *addr, size_t len)

munlock unlocks a range of the calling process' virtual pages.

munlock is the inverse of mlock and functions completely analogously to mlock, except that there is no EPERM failure.

— Function: int mlockall (int flags)

mlockall locks all the pages in a process' virtual memory address space, and/or any that are added to it in the future. This includes the pages of the code, data and stack segment, as well as shared libraries, user space kernel data, shared memory, and memory mapped files.

flags is a string of single bit flags represented by the following macros. They tell mlockall which of its functions you want. All other bits must be zero.

MCL_CURRENT
Lock all pages which currently exist in the calling process' virtual address space.
MCL_FUTURE
Set a mode such that any pages added to the process' virtual address space in the future will be locked from birth. This mode does not affect future address spaces owned by the same process so exec, which replaces a process' address space, wipes out MCL_FUTURE. See Executing a File.

When the function returns successfully, and you specified MCL_CURRENT, all of the process' pages are backed by (connected to) real frames (they are resident) and are marked to stay that way. This means the function may cause page-ins and have to wait for them.

When the process is in MCL_FUTURE mode because it successfully executed this function and specified MCL_CURRENT, any system call by the process that requires space be added to its virtual address space fails with errno = ENOMEM if locking the additional space would cause the process to exceed its locked page limit. In the case that the address space addition that can't be accommodated is stack expansion, the stack expansion fails and the kernel sends a SIGSEGV signal to the process.

When the function fails, it does not affect the lock status of any pages or the future locking mode.

The return value is zero if the function succeeds. Otherwise, it is -1 and errno is set accordingly. errno values specific to this function are:

ENOMEM
  • At least some of the specified address range does not exist in the calling process' virtual address space.
  • The locking would cause the process to exceed its locked page limit.

EPERM
The calling process is not superuser.
EINVAL
Undefined bits in flags are not zero.
ENOSYS
The kernel does not provide mlockall capability.

You can lock just specific pages with mlock. You unlock pages with munlockall and munlock.

— Function: int munlockall (void)

munlockall unlocks every page in the calling process' virtual address space and turn off MCL_FUTURE future locking mode.

The return value is zero if the function succeeds. Otherwise, it is -1 and errno is set accordingly. The only way this function can fail is for generic reasons that all functions and system calls can fail, so there are no specific errno values.

Next: , Previous: Memory, Up: Top

4 Character Handling

Programs that work with characters and strings often need to classify a character—is it alphabetic, is it a digit, is it whitespace, and so on—and perform case conversion operations on characters. The functions in the header file ctype.h are provided for this purpose. Since the choice of locale and character set can alter the classifications of particular character codes, all of these functions are affected by the current locale. (More precisely, they are affected by the locale currently selected for character classification—the LC_CTYPE category; see Locale Categories.)

The ISO C standard specifies two different sets of functions. The one set works on char type characters, the other one on wchar_t wide characters (see Extended Char Intro).

Next: , Up: Character Handling

4.1 Classification of Characters

This section explains the library functions for classifying characters. For example, isalpha is the function to test for an alphabetic character. It takes one argument, the character to test, and returns a nonzero integer if the character is alphabetic, and zero otherwise. You would use it like this: 

     if (isalpha (c))
       printf ("The character `%c' is alphabetic.\n", c);

Each of the functions in this section tests for membership in a particular class of characters; each has a name starting with `is'. Each of them takes one argument, which is a character to test, and returns an int which is treated as a boolean value. The character argument is passed as an int, and it may be the constant value EOF instead of a real character.

The attributes of any given character can vary between locales. See Locales, for more information on locales.

These functions are declared in the header file ctype.h.

— Function: int islower (int c)

Returns true if c is a lower-case letter. The letter need not be from the Latin alphabet, any alphabet representable is valid.

— Function: int isupper (int c)

Returns true if c is an upper-case letter. The letter need not be from the Latin alphabet, any alphabet representable is valid.

— Function: int isalpha (int c)

Returns true if c is an alphabetic character (a letter). If islower or isupper is true of a character, then isalpha is also true.

In some locales, there may be additional characters for which isalpha is true—letters which are neither upper case nor lower case. But in the standard "C" locale, there are no such additional characters.

— Function: int isdigit (int c)

Returns true if c is a decimal digit (`0' through `9').

— Function: int isalnum (int c)

Returns true if c is an alphanumeric character (a letter or number); in other words, if either isalpha or isdigit is true of a character, then isalnum is also true.

— Function: int isxdigit (int c)

Returns true if c is a hexadecimal digit. Hexadecimal digits include the normal decimal digits `0' through `9' and the letters `A' through `F' and `a' through `f'.

— Function: int ispunct (int c)

Returns true if c is a punctuation character. This means any printing character that is not alphanumeric or a space character.

— Function: int isspace (int c)

Returns true if c is a whitespace character. In the standard "C" locale, isspace returns true for only the standard whitespace characters:

' '
space
'\f'
formfeed
'\n'
newline
'\r'
carriage return
'\t'
horizontal tab
'\v'
vertical tab

— Function: int isblank (int c)

Returns true if c is a blank character; that is, a space or a tab. This function was originally a GNU extension, but was added in ISO C99.

— Function: int isgraph (int c)

Returns true if c is a graphic character; that is, a character that has a glyph associated with it. The whitespace characters are not considered graphic.

— Function: int isprint (int c)

Returns true if c is a printing character. Printing characters include all the graphic characters, plus the space (` ') character.

— Function: int iscntrl (int c)

Returns true if c is a control character (that is, a character that is not a printing character).

— Function: int isascii (int c)

Returns true if c is a 7-bit unsigned char value that fits into the US/UK ASCII character set. This function is a BSD extension and is also an SVID extension.

Next: , Previous: Classification of Characters, Up: Character Handling

4.2 Case Conversion

This section explains the library functions for performing conversions such as case mappings on characters. For example, toupper converts any character to upper case if possible. If the character can't be converted, toupper returns it unchanged.

These functions take one argument of type int, which is the character to convert, and return the converted character as an int. If the conversion is not applicable to the argument given, the argument is returned unchanged.

Compatibility Note: In pre-ISO C dialects, instead of returning the argument unchanged, these functions may fail when the argument is not suitable for the conversion. Thus for portability, you may need to write islower(c) ? toupper(c) : c rather than just toupper(c).

These functions are declared in the header file ctype.h.

— Function: int tolower (int c)

If c is an upper-case letter, tolower returns the corresponding lower-case letter. If c is not an upper-case letter, c is returned unchanged.

— Function: int toupper (int c)

If c is a lower-case letter, toupper returns the corresponding upper-case letter. Otherwise c is returned unchanged.

— Function: int toascii (int c)

This function converts c to a 7-bit unsigned char value that fits into the US/UK ASCII character set, by clearing the high-order bits. This function is a BSD extension and is also an SVID extension.

— Function: int _tolower (int c)

This is identical to tolower, and is provided for compatibility with the SVID. See SVID.

— Function: int _toupper (int c)

This is identical to toupper, and is provided for compatibility with the SVID.

Next: , Previous: Case Conversion, Up: Character Handling

4.3 Character class determination for wide characters

Amendment 1 to ISO C90 defines functions to classify wide characters. Although the original ISO C90 standard already defined the type wchar_t, no functions operating on them were defined.

The general design of the classification functions for wide characters is more general. It allows extensions to the set of available classifications, beyond those which are always available. The POSIX standard specifies how extensions can be made, and this is already implemented in the GNU C library implementation of the localedef program.

The character class functions are normally implemented with bitsets, with a bitset per character. For a given character, the appropriate bitset is read from a table and a test is performed as to whether a certain bit is set. Which bit is tested for is determined by the class.

For the wide character classification functions this is made visible. There is a type classification type defined, a function to retrieve this value for a given class, and a function to test whether a given character is in this class, using the classification value. On top of this the normal character classification functions as used for char objects can be defined.

— Data type: wctype_t

The wctype_t can hold a value which represents a character class. The only defined way to generate such a value is by using the wctype function.

This type is defined in wctype.h.

— Function: wctype_t wctype (const char *property)

The wctype returns a value representing a class of wide characters which is identified by the string property. Beside some standard properties each locale can define its own ones. In case no property with the given name is known for the current locale selected for the LC_CTYPE category, the function returns zero.

The properties known in every locale are:

"alnum" "alpha" "cntrl" "digit"
"graph" "lower" "print" "punct"
"space" "upper" "xdigit"

This function is declared in wctype.h.

To test the membership of a character to one of the non-standard classes the ISO C standard defines a completely new function.

— Function: int iswctype (wint_t wc, wctype_t desc)

This function returns a nonzero value if wc is in the character class specified by desc. desc must previously be returned by a successful call to wctype.

This function is declared in wctype.h.

To make it easier to use the commonly-used classification functions, they are defined in the C library. There is no need to use wctype if the property string is one of the known character classes. In some situations it is desirable to construct the property strings, and then it is important that wctype can also handle the standard classes.

— Function: int iswalnum (wint_t wc)

This function returns a nonzero value if wc is an alphanumeric character (a letter or number); in other words, if either iswalpha or iswdigit is true of a character, then iswalnum is also true.

This function can be implemented using 

          iswctype (wc, wctype ("alnum"))

It is declared in wctype.h.

— Function: int iswalpha (wint_t wc)

Returns true if wc is an alphabetic character (a letter). If iswlower or iswupper is true of a character, then iswalpha is also true.

In some locales, there may be additional characters for which iswalpha is true—letters which are neither upper case nor lower case. But in the standard "C" locale, there are no such additional characters.

This function can be implemented using 

          iswctype (wc, wctype ("alpha"))

It is declared in wctype.h.

— Function: int iswcntrl (wint_t wc)

Returns true if wc is a control character (that is, a character that is not a printing character).

This function can be implemented using 

          iswctype (wc, wctype ("cntrl"))

It is declared in wctype.h.

— Function: int iswdigit (wint_t wc)

Returns true if wc is a digit (e.g., `0' through `9'). Please note that this function does not only return a nonzero value for decimal digits, but for all kinds of digits. A consequence is that code like the following will not work unconditionally for wide characters: 

          n = 0;
          while (iswdigit (*wc))
            {
              n *= 10;
              n += *wc++ - L'0';
            }

This function can be implemented using 

          iswctype (wc, wctype ("digit"))

It is declared in wctype.h.

— Function: int iswgraph (wint_t wc)

Returns true if wc is a graphic character; that is, a character that has a glyph associated with it. The whitespace characters are not considered graphic.

This function can be implemented using 

          iswctype (wc, wctype ("graph"))

It is declared in wctype.h.

— Function: int iswlower (wint_t wc)

Returns true if wc is a lower-case letter. The letter need not be from the Latin alphabet, any alphabet representable is valid.

This function can be implemented using 

          iswctype (wc, wctype ("lower"))

It is declared in wctype.h.

— Function: int iswprint (wint_t wc)

Returns true if wc is a printing character. Printing characters include all the graphic characters, plus the space (` ') character.

This function can be implemented using 

          iswctype (wc, wctype ("print"))

It is declared in wctype.h.

— Function: int iswpunct (wint_t wc)

Returns true if wc is a punctuation character. This means any printing character that is not alphanumeric or a space character.

This function can be implemented using 

          iswctype (wc, wctype ("punct"))

It is declared in wctype.h.

— Function: int iswspace (wint_t wc)

Returns true if wc is a whitespace character. In the standard "C" locale, iswspace returns true for only the standard whitespace characters:

L' '
space
L'\f'
formfeed
L'\n'
newline
L'\r'
carriage return
L'\t'
horizontal tab
L'\v'
vertical tab

This function can be implemented using 

          iswctype (wc, wctype ("space"))

It is declared in wctype.h.

— Function: int iswupper (wint_t wc)

Returns true if wc is an upper-case letter. The letter need not be from the Latin alphabet, any alphabet representable is valid.

This function can be implemented using 

          iswctype (wc, wctype ("upper"))

It is declared in wctype.h.

— Function: int iswxdigit (wint_t wc)

Returns true if wc is a hexadecimal digit. Hexadecimal digits include the normal decimal digits `0' through `9' and the letters `A' through `F' and `a' through `f'.

This function can be implemented using 

          iswctype (wc, wctype ("xdigit"))

It is declared in wctype.h.

The GNU C library also provides a function which is not defined in the ISO C standard but which is available as a version for single byte characters as well.

— Function: int iswblank (wint_t wc)

Returns true if wc is a blank character; that is, a space or a tab. This function was originally a GNU extension, but was added in ISO C99. It is declared in wchar.h.

Next: , Previous: Classification of Wide Characters, Up: Character Handling

4.4 Notes on using the wide character classes

The first note is probably not astonishing but still occasionally a cause of problems. The iswXXX functions can be implemented using macros and in fact, the GNU C library does this. They are still available as real functions but when the wctype.h header is included the macros will be used. This is the same as the char type versions of these functions.

The second note covers something new. It can be best illustrated by a (real-world) example. The first piece of code is an excerpt from the original code. It is truncated a bit but the intention should be clear. 

     int
     is_in_class (int c, const char *class)
     {
       if (strcmp (class, "alnum") == 0)
         return isalnum (c);
       if (strcmp (class, "alpha") == 0)
         return isalpha (c);
       if (strcmp (class, "cntrl") == 0)
         return iscntrl (c);
       ...
       return 0;
     }

Now, with the wctype and iswctype you can avoid the if cascades, but rewriting the code as follows is wrong: 

     int
     is_in_class (int c, const char *class)
     {
       wctype_t desc = wctype (class);
       return desc ? iswctype ((wint_t) c, desc) : 0;
     }

The problem is that it is not guaranteed that the wide character representation of a single-byte character can be found using casting. In fact, usually this fails miserably. The correct solution to this problem is to write the code as follows: 

     int
     is_in_class (int c, const char *class)
     {
       wctype_t desc = wctype (class);
       return desc ? iswctype (btowc (c), desc) : 0;
     }

See Converting a Character, for more information on btowc. Note that this change probably does not improve the performance of the program a lot since the wctype function still has to make the string comparisons. It gets really interesting if the is_in_class function is called more than once for the same class name. In this case the variable desc could be computed once and reused for all the calls. Therefore the above form of the function is probably not the final one.

Previous: Using Wide Char Classes, Up: Character Handling

4.5 Mapping of wide characters.

The classification functions are also generalized by the ISO C standard. Instead of just allowing the two standard mappings, a locale can contain others. Again, the localedef program already supports generating such locale data files.

— Data Type: wctrans_t

This data type is defined as a scalar type which can hold a value representing the locale-dependent character mapping. There is no way to construct such a value apart from using the return value of the wctrans function.

This type is defined in wctype.h.

— Function: wctrans_t wctrans (const char *property)

The wctrans function has to be used to find out whether a named mapping is defined in the current locale selected for the LC_CTYPE category. If the returned value is non-zero, you can use it afterwards in calls to towctrans. If the return value is zero no such mapping is known in the current locale.

Beside locale-specific mappings there are two mappings which are guaranteed to be available in every locale:

"tolower" "toupper"

These functions are declared in wctype.h.

— Function: wint_t towctrans (wint_t wc, wctrans_t desc)

towctrans maps the input character wc according to the rules of the mapping for which desc is a descriptor, and returns the value it finds. desc must be obtained by a successful call to wctrans.

This function is declared in wctype.h.

For the generally available mappings, the ISO C standard defines convenient shortcuts so that it is not necessary to call wctrans for them.

— Function: wint_t towlower (wint_t wc)

If wc is an upper-case letter, towlower returns the corresponding lower-case letter. If wc is not an upper-case letter, wc is returned unchanged.

towlower can be implemented using 

          towctrans (wc, wctrans ("tolower"))

This function is declared in wctype.h.

— Function: wint_t towupper (wint_t wc)

If wc is a lower-case letter, towupper returns the corresponding upper-case letter. Otherwise wc is returned unchanged.

towupper can be implemented using 

          towctrans (wc, wctrans ("toupper"))

This function is declared in wctype.h.

The same warnings given in the last section for the use of the wide character classification functions apply here. It is not possible to simply cast a char type value to a wint_t and use it as an argument to towctrans calls.

Next: , Previous: Character Handling, Up: Top

5 String and Array Utilities

Operations on strings (or arrays of characters) are an important part of many programs. The GNU C library provides an extensive set of string utility functions, including functions for copying, concatenating, comparing, and searching strings. Many of these functions can also operate on arbitrary regions of storage; for example, the memcpy function can be used to copy the contents of any kind of array.

It's fairly common for beginning C programmers to “reinvent the wheel” by duplicating this functionality in their own code, but it pays to become familiar with the library functions and to make use of them, since this offers benefits in maintenance, efficiency, and portability.

For instance, you could easily compare one string to another in two lines of C code, but if you use the built-in strcmp function, you're less likely to make a mistake. And, since these library functions are typically highly optimized, your program may run faster too.

Next: , Up: String and Array Utilities

5.1 Representation of Strings

This section is a quick summary of string concepts for beginning C programmers. It describes how character strings are represented in C and some common pitfalls. If you are already familiar with this material, you can skip this section.

A string is an array of char objects. But string-valued variables are usually declared to be pointers of type char *. Such variables do not include space for the text of a string; that has to be stored somewhere else—in an array variable, a string constant, or dynamically allocated memory (see Memory Allocation). It's up to you to store the address of the chosen memory space into the pointer variable. Alternatively you can store a null pointer in the pointer variable. The null pointer does not point anywhere, so attempting to reference the string it points to gets an error.

“string” normally refers to multibyte character strings as opposed to wide character strings. Wide character strings are arrays of type wchar_t and as for multibyte character strings usually pointers of type wchar_t * are used.

By convention, a null character, '\0', marks the end of a multibyte character string and the null wide character, L'\0', marks the end of a wide character string. For example, in testing to see whether the char * variable p points to a null character marking the end of a string, you can write !*p or *p == '\0'.

A null character is quite different conceptually from a null pointer, although both are represented by the integer 0.

String literals appear in C program source as strings of characters between double-quote characters (`"') where the initial double-quote character is immediately preceded by a capital `L' (ell) character (as in L"foo"). In ISO C, string literals can also be formed by string concatenation: "a" "b" is the same as "ab". For wide character strings one can either use L"a" L"b" or L"a" "b". Modification of string literals is not allowed by the GNU C compiler, because literals are placed in read-only storage.

Character arrays that are declared const cannot be modified either. It's generally good style to declare non-modifiable string pointers to be of type const char *, since this often allows the C compiler to detect accidental modifications as well as providing some amount of documentation about what your program intends to do with the string.

The amount of memory allocated for the character array may extend past the null character that normally marks the end of the string. In this document, the term allocated size is always used to refer to the total amount of memory allocated for the string, while the term length refers to the number of characters up to (but not including) the terminating null character. A notorious source of program bugs is trying to put more characters in a string than fit in its allocated size. When writing code that extends strings or moves characters into a pre-allocated array, you should be very careful to keep track of the length of the text and make explicit checks for overflowing the array. Many of the library functions do not do this for you! Remember also that you need to allocate an extra byte to hold the null character that marks the end of the string.

Originally strings were sequences of bytes where each byte represents a single character. This is still true today if the strings are encoded using a single-byte character encoding. Things are different if the strings are encoded using a multibyte encoding (for more information on encodings see Extended Char Intro). There is no difference in the programming interface for these two kind of strings; the programmer has to be aware of this and interpret the byte sequences accordingly.

But since there is no separate interface taking care of these differences the byte-based string functions are sometimes hard to use. Since the count parameters of these functions specify bytes a call to strncpy could cut a multibyte character in the middle and put an incomplete (and therefore unusable) byte sequence in the target buffer.

To avoid these problems later versions of the ISO C standard introduce a second set of functions which are operating on wide characters (see Extended Char Intro). These functions don't have the problems the single-byte versions have since every wide character is a legal, interpretable value. This does not mean that cutting wide character strings at arbitrary points is without problems. It normally is for alphabet-based languages (except for non-normalized text) but languages based on syllables still have the problem that more than one wide character is necessary to complete a logical unit. This is a higher level problem which the C library functions are not designed to solve. But it is at least good that no invalid byte sequences can be created. Also, the higher level functions can also much easier operate on wide character than on multibyte characters so that a general advise is to use wide characters internally whenever text is more than simply copied.

The remaining of this chapter will discuss the functions for handling wide character strings in parallel with the discussion of the multibyte character strings since there is almost always an exact equivalent available.

Next: , Previous: Representation of Strings, Up: String and Array Utilities

5.2 String and Array Conventions

This chapter describes both functions that work on arbitrary arrays or blocks of memory, and functions that are specific to null-terminated arrays of characters and wide characters.

Functions that operate on arbitrary blocks of memory have names beginning with `mem' and `wmem' (such as memcpy and wmemcpy) and invariably take an argument which specifies the size (in bytes and wide characters respectively) of the block of memory to operate on. The array arguments and return values for these functions have type void * or wchar_t. As a matter of style, the elements of the arrays used with the `mem' functions are referred to as “bytes”. You can pass any kind of pointer to these functions, and the sizeof operator is useful in computing the value for the size argument. Parameters to the `wmem' functions must be of type wchar_t *. These functions are not really usable with anything but arrays of this type.

In contrast, functions that operate specifically on strings and wide character strings have names beginning with `str' and `wcs' respectively (such as strcpy and wcscpy) and look for a null character to terminate the string instead of requiring an explicit size argument to be passed. (Some of these functions accept a specified maximum length, but they also check for premature termination with a null character.) The array arguments and return values for these functions have type char * and wchar_t * respectively, and the array elements are referred to as “characters” and “wide characters”.

In many cases, there are both `mem' and `str'/`wcs' versions of a function. The one that is more appropriate to use depends on the exact situation. When your program is manipulating arbitrary arrays or blocks of storage, then you should always use the `mem' functions. On the other hand, when you are manipulating null-terminated strings it is usually more convenient to use the `str'/`wcs' functions, unless you already know the length of the string in advance. The `wmem' functions should be used for wide character arrays with known size.

Some of the memory and string functions take single characters as arguments. Since a value of type char is automatically promoted into an value of type int when used as a parameter, the functions are declared with int as the type of the parameter in question. In case of the wide character function the situation is similarly: the parameter type for a single wide character is wint_t and not wchar_t. This would for many implementations not be necessary since the wchar_t is large enough to not be automatically promoted, but since the ISO C standard does not require such a choice of types the wint_t type is used.

Next: , Previous: String/Array Conventions, Up: String and Array Utilities

5.3 String Length

You can get the length of a string using the strlen function. This function is declared in the header file string.h.

— Function: size_t strlen (const char *s)

The strlen function returns the length of the null-terminated string s in bytes. (In other words, it returns the offset of the terminating null character within the array.)

For example, 

          strlen ("hello, world")
              => 12

When applied to a character array, the strlen function returns the length of the string stored there, not its allocated size. You can get the allocated size of the character array that holds a string using the sizeof operator: 

          char string[32] = "hello, world";
          sizeof (string)
              => 32
          strlen (string)
              => 12

But beware, this will not work unless string is the character array itself, not a pointer to it. For example: 

          char string[32] = "hello, world";
          char *ptr = string;
          sizeof (string)
              => 32
          sizeof (ptr)
              => 4  /* (on a machine with 4 byte pointers) */

This is an easy mistake to make when you are working with functions that take string arguments; those arguments are always pointers, not arrays.

It must also be noted that for multibyte encoded strings the return value does not have to correspond to the number of characters in the string. To get this value the string can be converted to wide characters and wcslen can be used or something like the following code can be used: 

          /* The input is in string.
             The length is expected in n.  */
          {
            mbstate_t t;
            char *scopy = string;
            /* In initial state.  */
            memset (&t, '\0', sizeof (t));
            /* Determine number of characters.  */
            n = mbsrtowcs (NULL, &scopy, strlen (scopy), &t);
          }

This is cumbersome to do so if the number of characters (as opposed to bytes) is needed often it is better to work with wide characters.

The wide character equivalent is declared in wchar.h.

— Function: size_t wcslen (const wchar_t *ws)

The wcslen function is the wide character equivalent to strlen. The return value is the number of wide characters in the wide character string pointed to by ws (this is also the offset of the terminating null wide character of ws).

Since there are no multi wide character sequences making up one character the return value is not only the offset in the array, it is also the number of wide characters.

This function was introduced in Amendment 1 to ISO C90.

— Function: size_t strnlen (const char *s, size_t maxlen)

The strnlen function returns the length of the string s in bytes if this length is smaller than maxlen bytes. Otherwise it returns maxlen. Therefore this function is equivalent to (strlen (s) < n ? strlen (s) : maxlen) but it is more efficient and works even if the string s is not null-terminated. 

          char string[32] = "hello, world";
          strnlen (string, 32)
              => 12
          strnlen (string, 5)
              => 5

This function is a GNU extension and is declared in string.h.

— Function: size_t wcsnlen (const wchar_t *ws, size_t maxlen)

wcsnlen is the wide character equivalent to strnlen. The maxlen parameter specifies the maximum number of wide characters.

This function is a GNU extension and is declared in wchar.h.

Next: , Previous: String Length, Up: String and Array Utilities

5.4 Copying and Concatenation

You can use the functions described in this section to copy the contents of strings and arrays, or to append the contents of one string to another. The `str' and `mem' functions are declared in the header file string.h while the `wstr' and `wmem' functions are declared in the file wchar.h. A helpful way to remember the ordering of the arguments to the functions in this section is that it corresponds to an assignment expression, with the destination array specified to the left of the source array. All of these functions return the address of the destination array.

Most of these functions do not work properly if the source and destination arrays overlap. For example, if the beginning of the destination array overlaps the end of the source array, the original contents of that part of the source array may get overwritten before it is copied. Even worse, in the case of the string functions, the null character marking the end of the string may be lost, and the copy function might get stuck in a loop trashing all the memory allocated to your program.

All functions that have problems copying between overlapping arrays are explicitly identified in this manual. In addition to functions in this section, there are a few others like sprintf (see Formatted Output Functions) and scanf (see Formatted Input Functions).

— Function: void * memcpy (void *restrict to, const void *restrict from, size_t size)

The memcpy function copies size bytes from the object beginning at from into the object beginning at to. The behavior of this function is undefined if the two arrays to and from overlap; use memmove instead if overlapping is possible.

The value returned by memcpy is the value of to.

Here is an example of how you might use memcpy to copy the contents of an array: 

          struct foo *oldarray, *newarray;
          int arraysize;
          ...
          memcpy (new, old, arraysize * sizeof (struct foo));
— Function: wchar_t * wmemcpy (wchar_t *restrict wto, const wchar_t *restrict wfrom, size_t size)

The wmemcpy function copies size wide characters from the object beginning at wfrom into the object beginning at wto. The behavior of this function is undefined if the two arrays wto and wfrom overlap; use wmemmove instead if overlapping is possible.

The following is a possible implementation of wmemcpy but there are more optimizations possible. 

          wchar_t *
          wmemcpy (wchar_t *restrict wto, const wchar_t *restrict wfrom,
                   size_t size)
          {
            return (wchar_t *) memcpy (wto, wfrom, size * sizeof (wchar_t));
          }

The value returned by wmemcpy is the value of wto.

This function was introduced in Amendment 1 to ISO C90.

— Function: void * mempcpy (void *restrict to, const void *restrict from, size_t size)

The mempcpy function is nearly identical to the memcpy function. It copies size bytes from the object beginning at from into the object pointed to by to. But instead of returning the value of to it returns a pointer to the byte following the last written byte in the object beginning at to. I.e., the value is ((void *) ((char *) to + size)).

This function is useful in situations where a number of objects shall be copied to consecutive memory positions. 

          void *
          combine (void *o1, size_t s1, void *o2, size_t s2)
          {
            void *result = malloc (s1 + s2);
            if (result != NULL)
              mempcpy (mempcpy (result, o1, s1), o2, s2);
            return result;
          }

This function is a GNU extension.

— Function: wchar_t * wmempcpy (wchar_t *restrict wto, const wchar_t *restrict wfrom, size_t size)

The wmempcpy function is nearly identical to the wmemcpy function. It copies size wide characters from the object beginning at wfrom into the object pointed to by wto. But instead of returning the value of wto it returns a pointer to the wide character following the last written wide character in the object beginning at wto. I.e., the value is wto + size.

This function is useful in situations where a number of objects shall be copied to consecutive memory positions.

The following is a possible implementation of wmemcpy but there are more optimizations possible. 

          wchar_t *
          wmempcpy (wchar_t *restrict wto, const wchar_t *restrict wfrom,
                    size_t size)
          {
            return (wchar_t *) mempcpy (wto, wfrom, size * sizeof (wchar_t));
          }

This function is a GNU extension.

— Function: void * memmove (void *to, const void *from, size_t size)

memmove copies the size bytes at from into the size bytes at to, even if those two blocks of space overlap. In the case of overlap, memmove is careful to copy the original values of the bytes in the block at from, including those bytes which also belong to the block at to.

The value returned by memmove is the value of to.

— Function: wchar_t * wmemmove (wchar *wto, const wchar_t *wfrom, size_t size)

wmemmove copies the size wide characters at wfrom into the size wide characters at wto, even if those two blocks of space overlap. In the case of overlap, memmove is careful to copy the original values of the wide characters in the block at wfrom, including those wide characters which also belong to the block at wto.

The following is a possible implementation of wmemcpy but there are more optimizations possible. 

          wchar_t *
          wmempcpy (wchar_t *restrict wto, const wchar_t *restrict wfrom,
                    size_t size)
          {
            return (wchar_t *) mempcpy (wto, wfrom, size * sizeof (wchar_t));
          }

The value returned by wmemmove is the value of wto.

This function is a GNU extension.

— Function: void * memccpy (void *restrict to, const void *restrict from, int c, size_t size)

This function copies no more than size bytes from from to to, stopping if a byte matching c is found. The return value is a pointer into to one byte past where c was copied, or a null pointer if no byte matching c appeared in the first size bytes of from.

— Function: void * memset (void *block, int c, size_t size)

This function copies the value of c (converted to an unsigned char) into each of the first size bytes of the object beginning at block. It returns the value of block.

— Function: wchar_t * wmemset (wchar_t *block, wchar_t wc, size_t size)

This function copies the value of wc into each of the first size wide characters of the object beginning at block. It returns the value of block.

— Function: char * strcpy (char *restrict to, const char *restrict from)

This copies characters from the string from (up to and including the terminating null character) into the string to. Like memcpy, this function has undefined results if the strings overlap. The return value is the value of to.

— Function: wchar_t * wcscpy (wchar_t *restrict wto, const wchar_t *restrict wfrom)

This copies wide characters from the string wfrom (up to and including the terminating null wide character) into the string wto. Like wmemcpy, this function has undefined results if the strings overlap. The return value is the value of wto.

— Function: char * strncpy (char *restrict to, const char *restrict from, size_t size)

This function is similar to strcpy but always copies exactly size characters into to.

If the length of from is more than size, then strncpy copies just the first size characters. Note that in this case there is no null terminator written into to.

If the length of from is less than size, then strncpy copies all of from, followed by enough null characters to add up to size characters in all. This behavior is rarely useful, but it is specified by the ISO C standard.

The behavior of strncpy is undefined if the strings overlap.

Using strncpy as opposed to strcpy is a way to avoid bugs relating to writing past the end of the allocated space for to. However, it can also make your program much slower in one common case: copying a string which is probably small into a potentially large buffer. In this case, size may be large, and when it is, strncpy will waste a considerable amount of time copying null characters.

— Function: wchar_t * wcsncpy (wchar_t *restrict wto, const wchar_t *restrict wfrom, size_t size)

This function is similar to wcscpy but always copies exactly size wide characters into wto.

If the length of wfrom is more than size, then wcsncpy copies just the first size wide characters. Note that in this case there is no null terminator written into wto.

If the length of wfrom is less than size, then wcsncpy copies all of wfrom, followed by enough null wide characters to add up to size wide characters in all. This behavior is rarely useful, but it is specified by the ISO C standard.

The behavior of wcsncpy is undefined if the strings overlap.

Using wcsncpy as opposed to wcscpy is a way to avoid bugs relating to writing past the end of the allocated space for wto. However, it can also make your program much slower in one common case: copying a string which is probably small into a potentially large buffer. In this case, size may be large, and when it is, wcsncpy will waste a considerable amount of time copying null wide characters.

— Function: char * strdup (const char *s)

This function copies the null-terminated string s into a newly allocated string. The string is allocated using malloc; see Unconstrained Allocation. If malloc cannot allocate space for the new string, strdup returns a null pointer. Otherwise it returns a pointer to the new string.

— Function: wchar_t * wcsdup (const wchar_t *ws)

This function copies the null-terminated wide character string ws into a newly allocated string. The string is allocated using malloc; see Unconstrained Allocation. If malloc cannot allocate space for the new string, wcsdup returns a null pointer. Otherwise it returns a pointer to the new wide character string.

This function is a GNU extension.

— Function: char * strndup (const char *s, size_t size)

This function is similar to strdup but always copies at most size characters into the newly allocated string.

If the length of s is more than size, then strndup copies just the first size characters and adds a closing null terminator. Otherwise all characters are copied and the string is terminated.

This function is different to strncpy in that it always terminates the destination string.

strndup is a GNU extension.

— Function: char * stpcpy (char *restrict to, const char *restrict from)

This function is like strcpy, except that it returns a pointer to the end of the string to (that is, the address of the terminating null character to + strlen (from)) rather than the beginning.

For example, this program uses stpcpy to concatenate `foo' and `bar' to produce `foobar', which it then prints. 

          #include <string.h>
          #include <stdio.h>
          
          int
          main (void)
          {
            char buffer[10];
            char *to = buffer;
            to = stpcpy (to, "foo");
            to = stpcpy (to, "bar");
            puts (buffer);
            return 0;
          }

This function is not part of the ISO or POSIX standards, and is not customary on Unix systems, but we did not invent it either. Perhaps it comes from MS-DOG.

Its behavior is undefined if the strings overlap. The function is declared in string.h.

— Function: wchar_t * wcpcpy (wchar_t *restrict wto, const wchar_t *restrict wfrom)

This function is like wcscpy, except that it returns a pointer to the end of the string wto (that is, the address of the terminating null character wto + strlen (wfrom)) rather than the beginning.

This function is not part of ISO or POSIX but was found useful while developing the GNU C Library itself.

The behavior of wcpcpy is undefined if the strings overlap.

wcpcpy is a GNU extension and is declared in wchar.h.

— Function: char * stpncpy (char *restrict to, const char *restrict from, size_t size)

This function is similar to stpcpy but copies always exactly size characters into to.

If the length of from is more then size, then stpncpy copies just the first size characters and returns a pointer to the character directly following the one which was copied last. Note that in this case there is no null terminator written into to.

If the length of from is less than size, then stpncpy copies all of from, followed by enough null characters to add up to size characters in all. This behavior is rarely useful, but it is implemented to be useful in contexts where this behavior of the strncpy is used. stpncpy returns a pointer to the first written null character.

This function is not part of ISO or POSIX but was found useful while developing the GNU C Library itself.

Its behavior is undefined if the strings overlap. The function is declared in string.h.

— Function: wchar_t * wcpncpy (wchar_t *restrict wto, const wchar_t *restrict wfrom, size_t size)

This function is similar to wcpcpy but copies always exactly wsize characters into wto.

If the length of wfrom is more then size, then wcpncpy copies just the first size wide characters and returns a pointer to the wide character directly following the last non-null wide character which was copied last. Note that in this case there is no null terminator written into wto.

If the length of wfrom is less than size, then wcpncpy copies all of wfrom, followed by enough null characters to add up to size characters in all. This behavior is rarely useful, but it is implemented to be useful in contexts where this behavior of the wcsncpy is used. wcpncpy returns a pointer to the first written null character.

This function is not part of ISO or POSIX but was found useful while developing the GNU C Library itself.

Its behavior is undefined if the strings overlap.

wcpncpy is a GNU extension and is declared in wchar.h.

— Macro: char * strdupa (const char *s)

This macro is similar to strdup but allocates the new string using alloca instead of malloc (see Variable Size Automatic). This means of course the returned string has the same limitations as any block of memory allocated using alloca.

For obvious reasons strdupa is implemented only as a macro; you cannot get the address of this function. Despite this limitation it is a useful function. The following code shows a situation where using malloc would be a lot more expensive. 

          #include <paths.h>
          #include <string.h>
          #include <stdio.h>
          
          const char path[] = _PATH_STDPATH;
          
          int
          main (void)
          {
            char *wr_path = strdupa (path);
            char *cp = strtok (wr_path, ":");
          
            while (cp != NULL)
              {
                puts (cp);
                cp = strtok (NULL, ":");
              }
            return 0;
          }

Please note that calling strtok using path directly is invalid. It is also not allowed to call strdupa in the argument list of strtok since strdupa uses alloca (see Variable Size Automatic) can interfere with the parameter passing.

This function is only available if GNU CC is used.

— Macro: char * strndupa (const char *s, size_t size)

This function is similar to strndup but like strdupa it allocates the new string using alloca see Variable Size Automatic. The same advantages and limitations of strdupa are valid for strndupa, too.

This function is implemented only as a macro, just like strdupa. Just as strdupa this macro also must not be used inside the parameter list in a function call.

strndupa is only available if GNU CC is used.

— Function: char * strcat (char *restrict to, const char *restrict from)

The strcat function is similar to strcpy, except that the characters from from are concatenated or appended to the end of to, instead of overwriting it. That is, the first character from from overwrites the null character marking the end of to.

An equivalent definition for strcat would be: 

          char *
          strcat (char *restrict to, const char *restrict from)
          {
            strcpy (to + strlen (to), from);
            return to;
          }

This function has undefined results if the strings overlap.

— Function: wchar_t * wcscat (wchar_t *restrict wto, const wchar_t *restrict wfrom)

The wcscat function is similar to wcscpy, except that the characters from wfrom are concatenated or appended to the end of wto, instead of overwriting it. That is, the first character from wfrom overwrites the null character marking the end of wto.

An equivalent definition for wcscat would be: 

          wchar_t *
          wcscat (wchar_t *wto, const wchar_t *wfrom)
          {
            wcscpy (wto + wcslen (wto), wfrom);
            return wto;
          }

This function has undefined results if the strings overlap.

Programmers using the strcat or wcscat function (or the following strncat or wcsncar functions for that matter) can easily be recognized as lazy and reckless. In almost all situations the lengths of the participating strings are known (it better should be since how can one otherwise ensure the allocated size of the buffer is sufficient?) Or at least, one could know them if one keeps track of the results of the various function calls. But then it is very inefficient to use strcat/wcscat. A lot of time is wasted finding the end of the destination string so that the actual copying can start. This is a common example: 

     /* This function concatenates arbitrarily many strings.  The last
        parameter must be NULL.  */
     char *
     concat (const char *str, ...)
     {
       va_list ap, ap2;
       size_t total = 1;
       const char *s;
       char *result;
     
       va_start (ap, str);
       /* Actually va_copy, but this is the name more gcc versions
          understand.  */
       __va_copy (ap2, ap);
     
       /* Determine how much space we need.  */
       for (s = str; s != NULL; s = va_arg (ap, const char *))
         total += strlen (s);
     
       va_end (ap);
     
       result = (char *) malloc (total);
       if (result != NULL)
         {
           result[0] = '\0';
     
           /* Copy the strings.  */
           for (s = str; s != NULL; s = va_arg (ap2, const char *))
             strcat (result, s);
         }
     
       va_end (ap2);
     
       return result;
     }

This looks quite simple, especially the second loop where the strings are actually copied. But these innocent lines hide a major performance penalty. Just imagine that ten strings of 100 bytes each have to be concatenated. For the second string we search the already stored 100 bytes for the end of the string so that we can append the next string. For all strings in total the comparisons necessary to find the end of the intermediate results sums up to 5500! If we combine the copying with the search for the allocation we can write this function more efficient: 

     char *
     concat (const char *str, ...)
     {
       va_list ap;
       size_t allocated = 100;
       char *result = (char *) malloc (allocated);
     
       if (result != NULL)
         {
           char *newp;
           char *wp;
     
           va_start (ap, str);
     
           wp = result;
           for (s = str; s != NULL; s = va_arg (ap, const char *))
             {
               size_t len = strlen (s);
     
               /* Resize the allocated memory if necessary.  */
               if (wp + len + 1 > result + allocated)
                 {
                   allocated = (allocated + len) * 2;
                   newp = (char *) realloc (result, allocated);
                   if (newp == NULL)
                     {
                       free (result);
                       return NULL;
                     }
                   wp = newp + (wp - result);
                   result = newp;
                 }
     
               wp = mempcpy (wp, s, len);
             }
     
           /* Terminate the result string.  */
           *wp++ = '\0';
     
           /* Resize memory to the optimal size.  */
           newp = realloc (result, wp - result);
           if (newp != NULL)
             result = newp;
     
           va_end (ap);
         }
     
       return result;
     }

With a bit more knowledge about the input strings one could fine-tune the memory allocation. The difference we are pointing to here is that we don't use strcat anymore. We always keep track of the length of the current intermediate result so we can safe us the search for the end of the string and use mempcpy. Please note that we also don't use stpcpy which might seem more natural since we handle with strings. But this is not necessary since we already know the length of the string and therefore can use the faster memory copying function. The example would work for wide characters the same way.

Whenever a programmer feels the need to use strcat she or he should think twice and look through the program whether the code cannot be rewritten to take advantage of already calculated results. Again: it is almost always unnecessary to use strcat.

— Function: char * strncat (char *restrict to, const char *restrict from, size_t size)

This function is like strcat except that not more than size characters from from are appended to the end of to. A single null character is also always appended to to, so the total allocated size of to must be at least size + 1 bytes longer than its initial length.

The strncat function could be implemented like this: 

          char *
          strncat (char *to, const char *from, size_t size)
          {
            to[strlen (to) + size] = '\0';
            strncpy (to + strlen (to), from, size);
            return to;
          }

The behavior of strncat is undefined if the strings overlap.

— Function: wchar_t * wcsncat (wchar_t *restrict wto, const wchar_t *restrict wfrom, size_t size)

This function is like wcscat except that not more than size characters from from are appended to the end of to. A single null character is also always appended to to, so the total allocated size of to must be at least size + 1 bytes longer than its initial length.

The wcsncat function could be implemented like this: 

          wchar_t *
          wcsncat (wchar_t *restrict wto, const wchar_t *restrict wfrom,
                   size_t size)
          {
            wto[wcslen (to) + size] = L'\0';
            wcsncpy (wto + wcslen (wto), wfrom, size);
            return wto;
          }

The behavior of wcsncat is undefined if the strings overlap.

Here is an example showing the use of strncpy and strncat (the wide character version is equivalent). Notice how, in the call to strncat, the size parameter is computed to avoid overflowing the character array buffer. 

     #include <string.h>
     #include <stdio.h>
     
     #define SIZE 10
     
     static char buffer[SIZE];
     
     main ()
     {
       strncpy (buffer, "hello", SIZE);
       puts (buffer);
       strncat (buffer, ", world", SIZE - strlen (buffer) - 1);
       puts (buffer);
     }

The output produced by this program looks like: 

     hello
     hello, wo
— Function: void bcopy (const void *from, void *to, size_t size)

This is a partially obsolete alternative for memmove, derived from BSD. Note that it is not quite equivalent to memmove, because the arguments are not in the same order and there is no return value.

— Function: void bzero (void *block, size_t size)

This is a partially obsolete alternative for memset, derived from BSD. Note that it is not as general as memset, because the only value it can store is zero.

Next: , Previous: Copying and Concatenation, Up: String and Array Utilities

5.5 String/Array Comparison

You can use the functions in this section to perform comparisons on the contents of strings and arrays. As well as checking for equality, these functions can also be used as the ordering functions for sorting operations. See Searching and Sorting, for an example of this.

Unlike most comparison operations in C, the string comparison functions return a nonzero value if the strings are not equivalent rather than if they are. The sign of the value indicates the relative ordering of the first characters in the strings that are not equivalent: a negative value indicates that the first string is “less” than the second, while a positive value indicates that the first string is “greater”.

The most common use of these functions is to check only for equality. This is canonically done with an expression like `! strcmp (s1, s2)'.

All of these functions are declared in the header file string.h.

— Function: int memcmp (const void *a1, const void *a2, size_t size)

The function memcmp compares the size bytes of memory beginning at a1 against the size bytes of memory beginning at a2. The value returned has the same sign as the difference between the first differing pair of bytes (interpreted as unsigned char objects, then promoted to int).

If the contents of the two blocks are equal, memcmp returns 0.

— Function: int wmemcmp (const wchar_t *a1, const wchar_t *a2, size_t size)

The function wmemcmp compares the size wide characters beginning at a1 against the size wide characters beginning at a2. The value returned is smaller than or larger than zero depending on whether the first differing wide character is a1 is smaller or larger than the corresponding character in a2.

If the contents of the two blocks are equal, wmemcmp returns 0.

On arbitrary arrays, the memcmp function is mostly useful for testing equality. It usually isn't meaningful to do byte-wise ordering comparisons on arrays of things other than bytes. For example, a byte-wise comparison on the bytes that make up floating-point numbers isn't likely to tell you anything about the relationship between the values of the floating-point numbers.

wmemcmp is really only useful to compare arrays of type wchar_t since the function looks at sizeof (wchar_t) bytes at a time and this number of bytes is system dependent.

You should also be careful about using memcmp to compare objects that can contain “holes”, such as the padding inserted into structure objects to enforce alignment requirements, extra space at the end of unions, and extra characters at the ends of strings whose length is less than their allocated size. The contents of these “holes” are indeterminate and may cause strange behavior when performing byte-wise comparisons. For more predictable results, perform an explicit component-wise comparison.

For example, given a structure type definition like: 

     struct foo
       {
         unsigned char tag;
         union
           {
             double f;
             long i;
             char *p;
           } value;
       };

you are better off writing a specialized comparison function to compare struct foo objects instead of comparing them with memcmp.

— Function: int strcmp (const char *s1, const char *s2)

The strcmp function compares the string s1 against s2, returning a value that has the same sign as the difference between the first differing pair of characters (interpreted as unsigned char objects, then promoted to int).

If the two strings are equal, strcmp returns 0.

A consequence of the ordering used by strcmp is that if s1 is an initial substring of s2, then s1 is considered to be “less than” s2.

strcmp does not take sorting conventions of the language the strings are written in into account. To get that one has to use strcoll.

— Function: int wcscmp (const wchar_t *ws1, const wchar_t *ws2)

The wcscmp function compares the wide character string ws1 against ws2. The value returned is smaller than or larger than zero depending on whether the first differing wide character is ws1 is smaller or larger than the corresponding character in ws2.

If the two strings are equal, wcscmp returns 0.

A consequence of the ordering used by wcscmp is that if ws1 is an initial substring of ws2, then ws1 is considered to be “less than” ws2.

wcscmp does not take sorting conventions of the language the strings are written in into account. To get that one has to use wcscoll.

— Function: int strcasecmp (const char *s1, const char *s2)

This function is like strcmp, except that differences in case are ignored. How uppercase and lowercase characters are related is determined by the currently selected locale. In the standard "C" locale the characters Ä and ä do not match but in a locale which regards these characters as parts of the alphabet they do match.

strcasecmp is derived from BSD.

— Function: int wcscasecmp (const wchar_t *ws1, const wchar_T *ws2)

This function is like wcscmp, except that differences in case are ignored. How uppercase and lowercase characters are related is determined by the currently selected locale. In the standard "C" locale the characters Ä and ä do not match but in a locale which regards these characters as parts of the alphabet they do match.

wcscasecmp is a GNU extension.

— Function: int strncmp (const char *s1, const char *s2, size_t size)

This function is the similar to strcmp, except that no more than size characters are compared. In other words, if the two strings are the same in their first size characters, the return value is zero.

— Function: int wcsncmp (const wchar_t *ws1, const wchar_t *ws2, size_t size)

This function is the similar to wcscmp, except that no more than size wide characters are compared. In other words, if the two strings are the same in their first size wide characters, the return value is zero.

— Function: int strncasecmp (const char *s1, const char *s2, size_t n)

This function is like strncmp, except that differences in case are ignored. Like strcasecmp, it is locale dependent how uppercase and lowercase characters are related.

strncasecmp is a GNU extension.

— Function: int wcsncasecmp (const wchar_t *ws1, const wchar_t *s2, size_t n)

This function is like wcsncmp, except that differences in case are ignored. Like wcscasecmp, it is locale dependent how uppercase and lowercase characters are related.

wcsncasecmp is a GNU extension.

Here are some examples showing the use of strcmp and strncmp (equivalent examples can be constructed for the wide character functions). These examples assume the use of the ASCII character set. (If some other character set—say, EBCDIC—is used instead, then the glyphs are associated with different numeric codes, and the return values and ordering may differ.) 

     strcmp ("hello", "hello")
         => 0    /* These two strings are the same. */
     strcmp ("hello", "Hello")
         => 32   /* Comparisons are case-sensitive. */
     strcmp ("hello", "world")
         => -15  /* The character 'h' comes before 'w'. */
     strcmp ("hello", "hello, world")
         => -44  /* Comparing a null character against a comma. */
     strncmp ("hello", "hello, world", 5)
         => 0    /* The initial 5 characters are the same. */
     strncmp ("hello, world", "hello, stupid world!!!", 5)
         => 0    /* The initial 5 characters are the same. */
— Function: int strverscmp (const char *s1, const char *s2)

The strverscmp function compares the string s1 against s2, considering them as holding indices/version numbers. Return value follows the same conventions as found in the strverscmp function. In fact, if s1 and s2 contain no digits, strverscmp behaves like strcmp.

Basically, we compare strings normally (character by character), until we find a digit in each string - then we enter a special comparison mode, where each sequence of digits is taken as a whole. If we reach the end of these two parts without noticing a difference, we return to the standard comparison mode. There are two types of numeric parts: "integral" and "fractional" (those begin with a '0'). The types of the numeric parts affect the way we sort them: 

          strverscmp ("no digit", "no digit")
              => 0    /* same behavior as strcmp. */
          strverscmp ("item#99", "item#100")
              => <0   /* same prefix, but 99 < 100. */
          strverscmp ("alpha1", "alpha001")
              => >0   /* fractional part inferior to integral one. */
          strverscmp ("part1_f012", "part1_f01")
              => >0   /* two fractional parts. */
          strverscmp ("foo.009", "foo.0")
              => <0   /* idem, but with leading zeroes only. */

This function is especially useful when dealing with filename sorting, because filenames frequently hold indices/version numbers.

strverscmp is a GNU extension.

— Function: int bcmp (const void *a1, const void *a2, size_t size)

This is an obsolete alias for memcmp, derived from BSD.

Next: , Previous: String/Array Comparison, Up: String and Array Utilities

5.6 Collation Functions

In some locales, the conventions for lexicographic ordering differ from the strict numeric ordering of character codes. For example, in Spanish most glyphs with diacritical marks such as accents are not considered distinct letters for the purposes of collation. On the other hand, the two-character sequence `ll' is treated as a single letter that is collated immediately after `l'.

You can use the functions strcoll and strxfrm (declared in the headers file string.h) and wcscoll and wcsxfrm (declared in the headers file wchar) to compare strings using a collation ordering appropriate for the current locale. The locale used by these functions in particular can be specified by setting the locale for the LC_COLLATE category; see Locales. In the standard C locale, the collation sequence for strcoll is the same as that for strcmp. Similarly, wcscoll and wcscmp are the same in this situation.

Effectively, the way these functions work is by applying a mapping to transform the characters in a string to a byte sequence that represents the string's position in the collating sequence of the current locale. Comparing two such byte sequences in a simple fashion is equivalent to comparing the strings with the locale's collating sequence.

The functions strcoll and wcscoll perform this translation implicitly, in order to do one comparison. By contrast, strxfrm and wcsxfrm perform the mapping explicitly. If you are making multiple comparisons using the same string or set of strings, it is likely to be more efficient to use strxfrm or wcsxfrm to transform all the strings just once, and subsequently compare the transformed strings with strcmp or wcscmp.

— Function: int strcoll (const char *s1, const char *s2)

The strcoll function is similar to strcmp but uses the collating sequence of the current locale for collation (the LC_COLLATE locale).

— Function: int wcscoll (const wchar_t *ws1, const wchar_t *ws2)

The wcscoll function is similar to wcscmp but uses the collating sequence of the current locale for collation (the LC_COLLATE locale).

Here is an example of sorting an array of strings, using strcoll to compare them. The actual sort algorithm is not written here; it comes from qsort (see Array Sort Function). The job of the code shown here is to say how to compare the strings while sorting them. (Later on in this section, we will show a way to do this more efficiently using strxfrm.) 

     /* This is the comparison function used with qsort. */
     
     int
     compare_elements (char **p1, char **p2)
     {
       return strcoll (*p1, *p2);
     }
     
     /* This is the entry point---the function to sort
        strings using the locale's collating sequence. */
     
     void
     sort_strings (char **array, int nstrings)
     {
       /* Sort temp_array by comparing the strings. */
       qsort (array, nstrings,
              sizeof (char *), compare_elements);
     }

— Function: size_t strxfrm (char *restrict to, const char *restrict from, size_t size)

The function strxfrm transforms the string from using the collation transformation determined by the locale currently selected for collation, and stores the transformed string in the array to. Up to size characters (including a terminating null character) are stored.

The behavior is undefined if the strings to and from overlap; see Copying and Concatenation.

The return value is the length of the entire transformed string. This value is not affected by the value of size, but if it is greater or equal than size, it means that the transformed string did not entirely fit in the array to. In this case, only as much of the string as actually fits was stored. To get the whole transformed string, call strxfrm again with a bigger output array.

The transformed string may be longer than the original string, and it may also be shorter.

If size is zero, no characters are stored in to. In this case, strxfrm simply returns the number of characters that would be the length of the transformed string. This is useful for determining what size the allocated array should be. It does not matter what to is if size is zero; to may even be a null pointer.

— Function: size_t wcsxfrm (wchar_t *restrict wto, const wchar_t *wfrom, size_t size)

The function wcsxfrm transforms wide character string wfrom using the collation transformation determined by the locale currently selected for collation, and stores the transformed string in the array wto. Up to size wide characters (including a terminating null character) are stored.

The behavior is undefined if the strings wto and wfrom overlap; see Copying and Concatenation.

The return value is the length of the entire transformed wide character string. This value is not affected by the value of size, but if it is greater or equal than size, it means that the transformed wide character string did not entirely fit in the array wto. In this case, only as much of the wide character string as actually fits was stored. To get the whole transformed wide character string, call wcsxfrm again with a bigger output array.

The transformed wide character string may be longer than the original wide character string, and it may also be shorter.

If size is zero, no characters are stored in to. In this case, wcsxfrm simply returns the number of wide characters that would be the length of the transformed wide character string. This is useful for determining what size the allocated array should be (remember to multiply with sizeof (wchar_t)). It does not matter what wto is if size is zero; wto may even be a null pointer.

Here is an example of how you can use strxfrm when you plan to do many comparisons. It does the same thing as the previous example, but much faster, because it has to transform each string only once, no matter how many times it is compared with other strings. Even the time needed to allocate and free storage is much less than the time we save, when there are many strings. 

     struct sorter { char *input; char *transformed; };
     
     /* This is the comparison function used with qsort
        to sort an array of struct sorter. */
     
     int
     compare_elements (struct sorter *p1, struct sorter *p2)
     {
       return strcmp (p1->transformed, p2->transformed);
     }
     
     /* This is the entry point---the function to sort
        strings using the locale's collating sequence. */
     
     void
     sort_strings_fast (char **array, int nstrings)
     {
       struct sorter temp_array[nstrings];
       int i;
     
       /* Set up temp_array.  Each element contains
          one input string and its transformed string. */
       for (i = 0; i < nstrings; i++)
         {
           size_t length = strlen (array[i]) * 2;
           char *transformed;
           size_t transformed_length;
     
           temp_array[i].input = array[i];
     
           /* First try a buffer perhaps big enough.  */
           transformed = (char *) xmalloc (length);
     
           /* Transform array[i].  */
           transformed_length = strxfrm (transformed, array[i], length);
     
           /* If the buffer was not large enough, resize it
              and try again.  */
           if (transformed_length >= length)
             {
               /* Allocate the needed space. +1 for terminating
                  NUL character.  */
               transformed = (char *) xrealloc (transformed,
                                                transformed_length + 1);
     
               /* The return value is not interesting because we know
                  how long the transformed string is.  */
               (void) strxfrm (transformed, array[i],
                               transformed_length + 1);
             }
     
           temp_array[i].transformed = transformed;
         }
     
       /* Sort temp_array by comparing transformed strings. */
       qsort (temp_array, sizeof (struct sorter),
              nstrings, compare_elements);
     
       /* Put the elements back in the permanent array
          in their sorted order. */
       for (i = 0; i < nstrings; i++)
         array[i] = temp_array[i].input;
     
       /* Free the strings we allocated. */
       for (i = 0; i < nstrings; i++)
         free (temp_array[i].transformed);
     }

The interesting part of this code for the wide character version would look like this: 

     void
     sort_strings_fast (wchar_t **array, int nstrings)
     {
       ...
           /* Transform array[i].  */
           transformed_length = wcsxfrm (transformed, array[i], length);
     
           /* If the buffer was not large enough, resize it
              and try again.  */
           if (transformed_length >= length)
             {
               /* Allocate the needed space. +1 for terminating
                  NUL character.  */
               transformed = (wchar_t *) xrealloc (transformed,
                                                   (transformed_length + 1)
                                                   * sizeof (wchar_t));
     
               /* The return value is not interesting because we know
                  how long the transformed string is.  */
               (void) wcsxfrm (transformed, array[i],
                               transformed_length + 1);
             }
       ...

Note the additional multiplication with sizeof (wchar_t) in the realloc call.

Compatibility Note: The string collation functions are a new feature of ISO C90. Older C dialects have no equivalent feature. The wide character versions were introduced in Amendment 1 to ISO C90.

Next: , Previous: Collation Functions, Up: String and Array Utilities

5.7 Search Functions

This section describes library functions which perform various kinds of searching operations on strings and arrays. These functions are declared in the header file string.h.

— Function: void * memchr (const void *block, int c, size_t size)

This function finds the first occurrence of the byte c (converted to an unsigned char) in the initial size bytes of the object beginning at block. The return value is a pointer to the located byte, or a null pointer if no match was found.

— Function: wchar_t * wmemchr (const wchar_t *block, wchar_t wc, size_t size)

This function finds the first occurrence of the wide character wc in the initial size wide characters of the object beginning at block. The return value is a pointer to the located wide character, or a null pointer if no match was found.

— Function: void * rawmemchr (const void *block, int c)

Often the memchr function is used with the knowledge that the byte c is available in the memory block specified by the parameters. But this means that the size parameter is not really needed and that the tests performed with it at runtime (to check whether the end of the block is reached) are not needed.

The rawmemchr function exists for just this situation which is surprisingly frequent. The interface is similar to memchr except that the size parameter is missing. The function will look beyond the end of the block pointed to by block in case the programmer made an error in assuming that the byte c is present in the block. In this case the result is unspecified. Otherwise the return value is a pointer to the located byte.

This function is of special interest when looking for the end of a string. Since all strings are terminated by a null byte a call like 

             rawmemchr (str, '\0')

will never go beyond the end of the string.

This function is a GNU extension.

— Function: void * memrchr (const void *block, int c, size_t size)

The function memrchr is like memchr, except that it searches backwards from the end of the block defined by block and size (instead of forwards from the front).

This function is a GNU extension.

— Function: char * strchr (const char *string, int c)

The strchr function finds the first occurrence of the character c (converted to a char) in the null-terminated string beginning at string. The return value is a pointer to the located character, or a null pointer if no match was found.

For example, 

          strchr ("hello, world", 'l')
              => "llo, world"
          strchr ("hello, world", '?')
              => NULL

The terminating null character is considered to be part of the string, so you can use this function get a pointer to the end of a string by specifying a null character as the value of the c argument. It would be better (but less portable) to use strchrnul in this case, though.

— Function: wchar_t * wcschr (const wchar_t *wstring, int wc)

The wcschr function finds the first occurrence of the wide character wc in the null-terminated wide character string beginning at wstring. The return value is a pointer to the located wide character, or a null pointer if no match was found.

The terminating null character is considered to be part of the wide character string, so you can use this function get a pointer to the end of a wide character string by specifying a null wude character as the value of the wc argument. It would be better (but less portable) to use wcschrnul in this case, though.

— Function: char * strchrnul (const char *string, int c)

strchrnul is the same as strchr except that if it does not find the character, it returns a pointer to string's terminating null character rather than a null pointer.

This function is a GNU extension.

— Function: wchar_t * wcschrnul (const wchar_t *wstring, wchar_t wc)

wcschrnul is the same as wcschr except that if it does not find the wide character, it returns a pointer to wide character string's terminating null wide character rather than a null pointer.

This function is a GNU extension.

One useful, but unusual, use of the strchr function is when one wants to have a pointer pointing to the NUL byte terminating a string. This is often written in this way: 

       s += strlen (s);

This is almost optimal but the addition operation duplicated a bit of the work already done in the strlen function. A better solution is this: 

       s = strchr (s, '\0');

There is no restriction on the second parameter of strchr so it could very well also be the NUL character. Those readers thinking very hard about this might now point out that the strchr function is more expensive than the strlen function since we have two abort criteria. This is right. But in the GNU C library the implementation of strchr is optimized in a special way so that strchr actually is faster.

— Function: char * strrchr (const char *string, int c)

The function strrchr is like strchr, except that it searches backwards from the end of the string string (instead of forwards from the front).

For example, 

          strrchr ("hello, world", 'l')
              => "ld"
— Function: wchar_t * wcsrchr (const wchar_t *wstring, wchar_t c)

The function wcsrchr is like wcschr, except that it searches backwards from the end of the string wstring (instead of forwards from the front).

— Function: char * strstr (const char *haystack, const char *needle)

This is like strchr, except that it searches haystack for a substring needle rather than just a single character. It returns a pointer into the string haystack that is the first character of the substring, or a null pointer if no match was found. If needle is an empty string, the function returns haystack.

For example, 

          strstr ("hello, world", "l")
              => "llo, world"
          strstr ("hello, world", "wo")
              => "world"
— Function: wchar_t * wcsstr (const wchar_t *haystack, const wchar_t *needle)

This is like wcschr, except that it searches haystack for a substring needle rather than just a single wide character. It returns a pointer into the string haystack that is the first wide character of the substring, or a null pointer if no match was found. If needle is an empty string, the function returns haystack.

— Function: wchar_t * wcswcs (const wchar_t *haystack, const wchar_t *needle)

wcswcs is an deprecated alias for wcsstr. This is the name originally used in the X/Open Portability Guide before the Amendment 1 to ISO C90 was published.

— Function: char * strcasestr (const char *haystack, const char *needle)

This is like strstr, except that it ignores case in searching for the substring. Like strcasecmp, it is locale dependent how uppercase and lowercase characters are related.

For example, 

          strcasestr ("hello, world", "L")
              => "llo, world"
          strcasestr ("hello, World", "wo")
              => "World"
— Function: void * memmem (const void *haystack, size_t haystack-len,
const void *needle, size_t needle-len)

This is like strstr, but needle and haystack are byte arrays rather than null-terminated strings. needle-len is the length of needle and haystack-len is the length of haystack.

This function is a GNU extension.

— Function: size_t strspn (const char *string, const char *skipset)

The strspn (“string span”) function returns the length of the initial substring of string that consists entirely of characters that are members of the set specified by the string skipset. The order of the characters in skipset is not important.

For example, 

          strspn ("hello, world", "abcdefghijklmnopqrstuvwxyz")
              => 5

Note that “character” is here used in the sense of byte. In a string using a multibyte character encoding (abstract) character consisting of more than one byte are not treated as an entity. Each byte is treated separately. The function is not locale-dependent.

— Function: size_t wcsspn (const wchar_t *wstring, const wchar_t *skipset)

The wcsspn (“wide character string span”) function returns the length of the initial substring of wstring that consists entirely of wide characters that are members of the set specified by the string skipset. The order of the wide characters in skipset is not important.

— Function: size_t strcspn (const char *string, const char *stopset)

The strcspn (“string complement span”) function returns the length of the initial substring of string that consists entirely of characters that are not members of the set specified by the string stopset. (In other words, it returns the offset of the first character in string that is a member of the set stopset.)

For example, 

          strcspn ("hello, world", " \t\n,.;!?")
              => 5

Note that “character” is here used in the sense of byte. In a string using a multibyte character encoding (abstract) character consisting of more than one byte are not treated as an entity. Each byte is treated separately. The function is not locale-dependent.

— Function: size_t wcscspn (const wchar_t *wstring, const wchar_t *stopset)

The wcscspn (“wide character string complement span”) function returns the length of the initial substring of wstring that consists entirely of wide characters that are not members of the set specified by the string stopset. (In other words, it returns the offset of the first character in string that is a member of the set stopset.)

— Function: char * strpbrk (const char *string, const char *stopset)

The strpbrk (“string pointer break”) function is related to strcspn, except that it returns a pointer to the first character in string that is a member of the set stopset instead of the length of the initial substring. It returns a null pointer if no such character from stopset is found.

For example, 

          strpbrk ("hello, world", " \t\n,.;!?")
              => ", world"

Note that “character” is here used in the sense of byte. In a string using a multibyte character encoding (abstract) character consisting of more than one byte are not treated as an entity. Each byte is treated separately. The function is not locale-dependent.

— Function: wchar_t * wcspbrk (const wchar_t *wstring, const wchar_t *stopset)

The wcspbrk (“wide character string pointer break”) function is related to wcscspn, except that it returns a pointer to the first wide character in wstring that is a member of the set stopset instead of the length of the initial substring. It returns a null pointer if no such character from stopset is found.

5.7.1 Compatibility String Search Functions

— Function: char * index (const char *string, int c)

index is another name for strchr; they are exactly the same. New code should always use strchr since this name is defined in ISO C while index is a BSD invention which never was available on System V derived systems.

— Function: char * rindex (const char *string, int c)

rindex is another name for strrchr; they are exactly the same. New code should always use strrchr since this name is defined in ISO C while rindex is a BSD invention which never was available on System V derived systems.

Next: , Previous: Search Functions, Up: String and Array Utilities

5.8 Finding Tokens in a String

It's fairly common for programs to have a need to do some simple kinds of lexical analysis and parsing, such as splitting a command string up into tokens. You can do this with the strtok function, declared in the header file string.h.

— Function: char * strtok (char *restrict newstring, const char *restrict delimiters)

A string can be split into tokens by making a series of calls to the function strtok.

The string to be split up is passed as the newstring argument on the first call only. The strtok function uses this to set up some internal state information. Subsequent calls to get additional tokens from the same string are indicated by passing a null pointer as the newstring argument. Calling strtok with another non-null newstring argument reinitializes the state information. It is guaranteed that no other library function ever calls strtok behind your back (which would mess up this internal state information).

The delimiters argument is a string that specifies a set of delimiters that may surround the token being extracted. All the initial characters that are members of this set are discarded. The first character that is not a member of this set of delimiters marks the beginning of the next token. The end of the token is found by looking for the next character that is a member of the delimiter set. This character in the original string newstring is overwritten by a null character, and the pointer to the beginning of the token in newstring is returned.

On the next call to strtok, the searching begins at the next character beyond the one that marked the end of the previous token. Note that the set of delimiters delimiters do not have to be the same on every call in a series of calls to strtok.

If the end of the string newstring is reached, or if the remainder of string consists only of delimiter characters, strtok returns a null pointer.

Note that “character” is here used in the sense of byte. In a string using a multibyte character encoding (abstract) character consisting of more than one byte are not treated as an entity. Each byte is treated separately. The function is not locale-dependent.

— Function: wchar_t * wcstok (wchar_t *newstring, const char *delimiters)

A string can be split into tokens by making a series of calls to the function wcstok.

The string to be split up is passed as the newstring argument on the first call only. The wcstok function uses this to set up some internal state information. Subsequent calls to get additional tokens from the same wide character string are indicated by passing a null pointer as the newstring argument. Calling wcstok with another non-null newstring argument reinitializes the state information. It is guaranteed that no other library function ever calls wcstok behind your back (which would mess up this internal state information).

The delimiters argument is a wide character string that specifies a set of delimiters that may surround the token being extracted. All the initial wide characters that are members of this set are discarded. The first wide character that is not a member of this set of delimiters marks the beginning of the next token. The end of the token is found by looking for the next wide character that is a member of the delimiter set. This wide character in the original wide character string newstring is overwritten by a null wide character, and the pointer to the beginning of the token in newstring is returned.

On the next call to wcstok, the searching begins at the next wide character beyond the one that marked the end of the previous token. Note that the set of delimiters delimiters do not have to be the same on every call in a series of calls to wcstok.

If the end of the wide character string newstring is reached, or if the remainder of string consists only of delimiter wide characters, wcstok returns a null pointer.

Note that “character” is here used in the sense of byte. In a string using a multibyte character encoding (abstract) character consisting of more than one byte are not treated as an entity. Each byte is treated separately. The function is not locale-dependent.

Warning: Since strtok and wcstok alter the string they is parsing, you should always copy the string to a temporary buffer before parsing it with strtok/wcstok (see Copying and Concatenation). If you allow strtok or wcstok to modify a string that came from another part of your program, you are asking for trouble; that string might be used for other purposes after strtok or wcstok has modified it, and it would not have the expected value.

The string that you are operating on might even be a constant. Then when strtok or wcstok tries to modify it, your program will get a fatal signal for writing in read-only memory. See Program Error Signals. Even if the operation of strtok or wcstok would not require a modification of the string (e.g., if there is exactly one token) the string can (and in the GNU libc case will) be modified.

This is a special case of a general principle: if a part of a program does not have as its purpose the modification of a certain data structure, then it is error-prone to modify the data structure temporarily.

The functions strtok and wcstok are not reentrant. See Nonreentrancy, for a discussion of where and why reentrancy is important.

Here is a simple example showing the use of strtok. 

     #include <string.h>
     #include <stddef.h>
     
     ...
     
     const char string[] = "words separated by spaces -- and, punctuation!";
     const char delimiters[] = " .,;:!-";
     char *token, *cp;
     
     ...
     
     cp = strdupa (string);                /* Make writable copy.  */
     token = strtok (cp, delimiters);      /* token => "words" */
     token = strtok (NULL, delimiters);    /* token => "separated" */
     token = strtok (NULL, delimiters);    /* token => "by" */
     token = strtok (NULL, delimiters);    /* token => "spaces" */
     token = strtok (NULL, delimiters);    /* token => "and" */
     token = strtok (NULL, delimiters);    /* token => "punctuation" */
     token = strtok (NULL, delimiters);    /* token => NULL */

The GNU C library contains two more functions for tokenizing a string which overcome the limitation of non-reentrancy. They are only available for multibyte character strings.

— Function: char * strtok_r (char *newstring, const char *delimiters, char **save_ptr)

Just like strtok, this function splits the string into several tokens which can be accessed by successive calls to strtok_r. The difference is that the information about the next token is stored in the space pointed to by the third argument, save_ptr, which is a pointer to a string pointer. Calling strtok_r with a null pointer for newstring and leaving save_ptr between the calls unchanged does the job without hindering reentrancy.

This function is defined in POSIX.1 and can be found on many systems which support multi-threading.

— Function: char * strsep (char **string_ptr, const char *delimiter)

This function has a similar functionality as strtok_r with the newstring argument replaced by the save_ptr argument. The initialization of the moving pointer has to be done by the user. Successive calls to strsep move the pointer along the tokens separated by delimiter, returning the address of the next token and updating string_ptr to point to the beginning of the next token.

One difference between strsep and strtok_r is that if the input string contains more than one character from delimiter in a row strsep returns an empty string for each pair of characters from delimiter. This means that a program normally should test for strsep returning an empty string before processing it.

This function was introduced in 4.3BSD and therefore is widely available.

Here is how the above example looks like when strsep is used. 

     #include <string.h>
     #include <stddef.h>
     
     ...
     
     const char string[] = "words separated by spaces -- and, punctuation!";
     const char delimiters[] = " .,;:!-";
     char *running;
     char *token;
     
     ...
     
     running = strdupa (string);
     token = strsep (&running, delimiters);    /* token => "words" */
     token = strsep (&running, delimiters);    /* token => "separated" */
     token = strsep (&running, delimiters);    /* token => "by" */
     token = strsep (&running, delimiters);    /* token => "spaces" */
     token = strsep (&running, delimiters);    /* token => "" */
     token = strsep (&running, delimiters);    /* token => "" */
     token = strsep (&running, delimiters);    /* token => "" */
     token = strsep (&running, delimiters);    /* token => "and" */
     token = strsep (&running, delimiters);    /* token => "" */
     token = strsep (&running, delimiters);    /* token => "punctuation" */
     token = strsep (&running, delimiters);    /* token => "" */
     token = strsep (&running, delimiters);    /* token => NULL */
— Function: char * basename (const char *filename)

The GNU version of the basename function returns the last component of the path in filename. This function is the preferred usage, since it does not modify the argument, filename, and respects trailing slashes. The prototype for basename can be found in string.h. Note, this function is overriden by the XPG version, if libgen.h is included.

Example of using GNU basename: 

          #include <string.h>
          
          int
          main (int argc, char *argv[])
          {
            char *prog = basename (argv[0]);
          
            if (argc < 2)
              {
                fprintf (stderr, "Usage %s <arg>\n", prog);
                exit (1);
              }
          
            ...
          }

Portability Note: This function may produce different results on different systems.

— Function: char * basename (char *path)

This is the standard XPG defined basename. It is similar in spirit to the GNU version, but may modify the path by removing trailing '/' characters. If the path is made up entirely of '/' characters, then "/" will be returned. Also, if path is NULL or an empty string, then "." is returned. The prototype for the XPG version can be found in libgen.h.

Example of using XPG basename: 

          #include <libgen.h>
          
          int
          main (int argc, char *argv[])
          {
            char *prog;
            char *path = strdupa (argv[0]);
          
            prog = basename (path);
          
            if (argc < 2)
              {
                fprintf (stderr, "Usage %s <arg>\n", prog);
                exit (1);
              }
          
            ...
          
          }
— Function: char * dirname (char *path)

The dirname function is the compliment to the XPG version of basename. It returns the parent directory of the file specified by path. If path is NULL, an empty string, or contains no '/' characters, then "." is returned. The prototype for this function can be found in libgen.h.

Next: , Previous: Finding Tokens in a String, Up: String and Array Utilities

5.9 strfry

The function below addresses the perennial programming quandary: “How do I take good data in string form and painlessly turn it into garbage?” This is actually a fairly simple task for C programmers who do not use the GNU C library string functions, but for programs based on the GNU C library, the strfry function is the preferred method for destroying string data.

The prototype for this function is in string.h.

— Function: char * strfry (char *string)

strfry creates a pseudorandom anagram of a string, replacing the input with the anagram in place. For each position in the string, strfry swaps it with a position in the string selected at random (from a uniform distribution). The two positions may be the same.

The return value of strfry is always string.

Portability Note: This function is unique to the GNU C library.

Next: , Previous: strfry, Up: String and Array Utilities

5.10 Trivial Encryption

The memfrob function converts an array of data to something unrecognizable and back again. It is not encryption in its usual sense since it is easy for someone to convert the encrypted data back to clear text. The transformation is analogous to Usenet's “Rot13” encryption method for obscuring offensive jokes from sensitive eyes and such. Unlike Rot13, memfrob works on arbitrary binary data, not just text. For true encryption, See Cryptographic Functions.

This function is declared in string.h.

— Function: void * memfrob (void *mem, size_t length)

memfrob transforms (frobnicates) each byte of the data structure at mem, which is length bytes long, by bitwise exclusive oring it with binary 00101010. It does the transformation in place and its return value is always mem.

Note that memfrob a second time on the same data structure returns it to its original state.

This is a good function for hiding information from someone who doesn't want to see it or doesn't want to see it very much. To really prevent people from retrieving the information, use stronger encryption such as that described in See Cryptographic Functions.

Portability Note: This function is unique to the GNU C library.

Next: , Previous: Trivial Encryption, Up: String and Array Utilities

5.11 Encode Binary Data

To store or transfer binary data in environments which only support text one has to encode the binary data by mapping the input bytes to characters in the range allowed for storing or transfering. SVID systems (and nowadays XPG compliant systems) provide minimal support for this task.

— Function: char * l64a (long int n)

This function encodes a 32-bit input value using characters from the basic character set. It returns a pointer to a 7 character buffer which contains an encoded version of n. To encode a series of bytes the user must copy the returned string to a destination buffer. It returns the empty string if n is zero, which is somewhat bizarre but mandated by the standard.
Warning: Since a static buffer is used this function should not be used in multi-threaded programs. There is no thread-safe alternative to this function in the C library.
Compatibility Note: The XPG standard states that the return value of l64a is undefined if n is negative. In the GNU implementation, l64a treats its argument as unsigned, so it will return a sensible encoding for any nonzero n; however, portable programs should not rely on this.

To encode a large buffer l64a must be called in a loop, once for each 32-bit word of the buffer. For example, one could do something like this: 

          char *
          encode (const void *buf, size_t len)
          {
            /* We know in advance how long the buffer has to be. */
            unsigned char *in = (unsigned char *) buf;
            char *out = malloc (6 + ((len + 3) / 4) * 6 + 1);
            char *cp = out, *p;
          
            /* Encode the length. */
            /* Using `htonl' is necessary so that the data can be
               decoded even on machines with different byte order.
               `l64a' can return a string shorter than 6 bytes, so 
               we pad it with encoding of 0 ('.') at the end by 
               hand. */
          
            p = stpcpy (cp, l64a (htonl (len)));
            cp = mempcpy (p, "......", 6 - (p - cp));
          
            while (len > 3)
              {
                unsigned long int n = *in++;
                n = (n << 8) | *in++;
                n = (n << 8) | *in++;
                n = (n << 8) | *in++;
                len -= 4;
                p = stpcpy (cp, l64a (htonl (n)));
                cp = mempcpy (p, "......", 6 - (p - cp));
              }
            if (len > 0)
              {
                unsigned long int n = *in++;
                if (--len > 0)
                  {
                    n = (n << 8) | *in++;
                    if (--len > 0)
                      n = (n << 8) | *in;
                  }
                cp = stpcpy (cp, l64a (htonl (n)));
              }
            *cp = '\0';
            return out;
          }

It is strange that the library does not provide the complete functionality needed but so be it.

To decode data produced with l64a the following function should be used.

— Function: long int a64l (const char *string)

The parameter string should contain a string which was produced by a call to l64a. The function processes at least 6 characters of this string, and decodes the characters it finds according to the table below. It stops decoding when it finds a character not in the table, rather like atoi; if you have a buffer which has been broken into lines, you must be careful to skip over the end-of-line characters.

The decoded number is returned as a long int value.

The l64a and a64l functions use a base 64 encoding, in which each character of an encoded string represents six bits of an input word. These symbols are used for the base 64 digits:

0 1 2 3 4 5 6 7
0 . / 0 1 2 3 4 5
8 6 7 8 9 A B C D
16 E F G H I J K L
24 M N O P Q R S T
32 U V W X Y Z a b
40 c d e f g h i j
48 k l m n o p q r
56 s t u v w x y z

This encoding scheme is not standard. There are some other encoding methods which are much more widely used (UU encoding, MIME encoding). Generally, it is better to use one of these encodings.

Previous: Encode Binary Data, Up: String and Array Utilities

5.12 Argz and Envz Vectors

argz vectors are vectors of strings in a contiguous block of memory, each element separated from its neighbors by null-characters ('\0').

Envz vectors are an extension of argz vectors where each element is a name-value pair, separated by a '=' character (as in a Unix environment).

Next: , Up: Argz and Envz Vectors

5.12.1 Argz Functions

Each argz vector is represented by a pointer to the first element, of type char *, and a size, of type size_t, both of which can be initialized to 0 to represent an empty argz vector. All argz functions accept either a pointer and a size argument, or pointers to them, if they will be modified.

The argz functions use malloc/realloc to allocate/grow argz vectors, and so any argz vector creating using these functions may be freed by using free; conversely, any argz function that may grow a string expects that string to have been allocated using malloc (those argz functions that only examine their arguments or modify them in place will work on any sort of memory). See Unconstrained Allocation.

All argz functions that do memory allocation have a return type of error_t, and return 0 for success, and ENOMEM if an allocation error occurs.

These functions are declared in the standard include file argz.h.

— Function: error_t argz_create (char *const argv[], char **argz, size_t *argz_len)

The argz_create function converts the Unix-style argument vector argv (a vector of pointers to normal C strings, terminated by (char *)0; see Program Arguments) into an argz vector with the same elements, which is returned in argz and argz_len.

— Function: error_t argz_create_sep (const char *string, int sep, char **argz, size_t *argz_len)

The argz_create_sep function converts the null-terminated string string into an argz vector (returned in argz and argz_len) by splitting it into elements at every occurrence of the character sep.

— Function: size_t argz_count (const char *argz, size_t arg_len)

Returns the number of elements in the argz vector argz and argz_len.

— Function: void argz_extract (char *argz, size_t argz_len, char **argv)

The argz_extract function converts the argz vector argz and argz_len into a Unix-style argument vector stored in argv, by putting pointers to every element in argz into successive positions in argv, followed by a terminator of 0. Argv must be pre-allocated with enough space to hold all the elements in argz plus the terminating (char *)0 ((argz_count (argz, argz_len) + 1) * sizeof (char *) bytes should be enough). Note that the string pointers stored into argv point into argz—they are not copies—and so argz must be copied if it will be changed while argv is still active. This function is useful for passing the elements in argz to an exec function (see Executing a File).

— Function: void argz_stringify (char *argz, size_t len, int sep)

The argz_stringify converts argz into a normal string with the elements separated by the character sep, by replacing each '\0' inside argz (except the last one, which terminates the string) with sep. This is handy for printing argz in a readable manner.

— Function: error_t argz_add (char **argz, size_t *argz_len, const char *str)

The argz_add function adds the string str to the end of the argz vector *argz, and updates *argz and *argz_len accordingly.

— Function: error_t argz_add_sep (char **argz, size_t *argz_len, const char *str, int delim)

The argz_add_sep function is similar to argz_add, but str is split into separate elements in the result at occurrences of the character delim. This is useful, for instance, for adding the components of a Unix search path to an argz vector, by using a value of ':' for delim.

— Function: error_t argz_append (char **argz, size_t *argz_len, const char *buf, size_t buf_len)

The argz_append function appends buf_len bytes starting at buf to the argz vector *argz, reallocating *argz to accommodate it, and adding buf_len to *argz_len.

— Function: void argz_delete (char **argz, size_t *argz_len, char *entry)

If entry points to the beginning of one of the elements in the argz vector *argz, the argz_delete function will remove this entry and reallocate *argz, modifying *argz and *argz_len accordingly. Note that as destructive argz functions usually reallocate their argz argument, pointers into argz vectors such as entry will then become invalid.

— Function: error_t argz_insert (char **argz, size_t *argz_len, char *before, const char *entry)

The argz_insert function inserts the string entry into the argz vector *argz at a point just before the existing element pointed to by before, reallocating *argz and updating *argz and *argz_len. If before is 0, entry is added to the end instead (as if by argz_add). Since the first element is in fact the same as *argz, passing in *argz as the value of before will result in entry being inserted at the beginning.

— Function: char * argz_next (char *argz, size_t argz_len, const char *entry)

The argz_next function provides a convenient way of iterating over the elements in the argz vector argz. It returns a pointer to the next element in argz after the element entry, or 0 if there are no elements following entry. If entry is 0, the first element of argz is returned.

This behavior suggests two styles of iteration: 

              char *entry = 0;
              while ((entry = argz_next (argz, argz_len, entry)))
                action;

(the double parentheses are necessary to make some C compilers shut up about what they consider a questionable while-test) and: 

              char *entry;
              for (entry = argz;
                   entry;
                   entry = argz_next (argz, argz_len, entry))
                action;

Note that the latter depends on argz having a value of 0 if it is empty (rather than a pointer to an empty block of memory); this invariant is maintained for argz vectors created by the functions here.

— Function: error_t argz_replace (char **argz, size_t *argz_len, const char *str, const char *with, unsigned *replace_count)

Replace any occurrences of the string str in argz with with, reallocating argz as necessary. If replace_count is non-zero, *replace_count will be incremented by number of replacements performed.

Previous: Argz Functions, Up: Argz and Envz Vectors

5.12.2 Envz Functions

Envz vectors are just argz vectors with additional constraints on the form of each element; as such, argz functions can also be used on them, where it makes sense.

Each element in an envz vector is a name-value pair, separated by a '=' character; if multiple '=' characters are present in an element, those after the first are considered part of the value, and treated like all other non-'\0' characters.

If no '=' characters are present in an element, that element is considered the name of a “null” entry, as distinct from an entry with an empty value: envz_get will return 0 if given the name of null entry, whereas an entry with an empty value would result in a value of ""; envz_entry will still find such entries, however. Null entries can be removed with envz_strip function.

As with argz functions, envz functions that may allocate memory (and thus fail) have a return type of error_t, and return either 0 or ENOMEM.

These functions are declared in the standard include file envz.h.

— Function: char * envz_entry (const char *envz, size_t envz_len, const char *name)

The envz_entry function finds the entry in envz with the name name, and returns a pointer to the whole entry—that is, the argz element which begins with name followed by a '=' character. If there is no entry with that name, 0 is returned.

— Function: char * envz_get (const char *envz, size_t envz_len, const char *name)

The envz_get function finds the entry in envz with the name name (like envz_entry), and returns a pointer to the value portion of that entry (following the '='). If there is no entry with that name (or only a null entry), 0 is returned.

— Function: error_t envz_add (char **envz, size_t *envz_len, const char *name, const char *value)

The envz_add function adds an entry to *envz (updating *envz and *envz_len) with the name name, and value value. If an entry with the same name already exists in envz, it is removed first. If value is 0, then the new entry will the special null type of entry (mentioned above).

— Function: error_t envz_merge (char **envz, size_t *envz_len, const char *envz2, size_t envz2_len, int override)

The envz_merge function adds each entry in envz2 to envz, as if with envz_add, updating *envz and *envz_len. If override is true, then values in envz2 will supersede those with the same name in envz, otherwise not.

Null entries are treated just like other entries in this respect, so a null entry in envz can prevent an entry of the same name in envz2 from being added to envz, if override is false.

— Function: void envz_strip (char **envz, size_t *envz_len)

The envz_strip function removes any null entries from envz, updating *envz and *envz_len.

Next: , Previous: String and Array Utilities, Up: Top

6 Character Set Handling

Character sets used in the early days of computing had only six, seven, or eight bits for each character: there was never a case where more than eight bits (one byte) were used to represent a single character. The limitations of this approach became more apparent as more people grappled with non-Roman character sets, where not all the characters that make up a language's character set can be represented by 2^8 choices. This chapter shows the functionality that was added to the C library to support multiple character sets.

Next: , Up: Character Set Handling

6.1 Introduction to Extended Characters

A variety of solutions is available to overcome the differences between character sets with a 1:1 relation between bytes and characters and character sets with ratios of 2:1 or 4:1. The remainder of this section gives a few examples to help understand the design decisions made while developing the functionality of the C library.

A distinction we have to make right away is between internal and external representation. Internal representation means the representation used by a program while keeping the text in memory. External representations are used when text is stored or transmitted through some communication channel. Examples of external representations include files waiting in a directory to be read and parsed.

Traditionally there has been no difference between the two representations. It was equally comfortable and useful to use the same single-byte representation internally and externally. This comfort level decreases with more and larger character sets.

One of the problems to overcome with the internal representation is handling text that is externally encoded using different character sets. Assume a program that reads two texts and compares them using some metric. The comparison can be usefully done only if the texts are internally kept in a common format.

For such a common format (= character set) eight bits are certainly no longer enough. So the smallest entity will have to grow: wide characters will now be used. Instead of one byte per character, two or four will be used instead. (Three are not good to address in memory and more than four bytes seem not to be necessary).

As shown in some other part of this manual, a completely new family has been created of functions that can handle wide character texts in memory. The most commonly used character sets for such internal wide character representations are Unicode and ISO 10646 (also known as UCS for Universal Character Set). Unicode was originally planned as a 16-bit character set; whereas, ISO 10646 was designed to be a 31-bit large code space. The two standards are practically identical. They have the same character repertoire and code table, but Unicode specifies added semantics. At the moment, only characters in the first 0x10000 code positions (the so-called Basic Multilingual Plane, BMP) have been assigned, but the assignment of more specialized characters outside this 16-bit space is already in progress. A number of encodings have been defined for Unicode and ISO 10646 characters: UCS-2 is a 16-bit word that can only represent characters from the BMP, UCS-4 is a 32-bit word than can represent any Unicode and ISO 10646 character, UTF-8 is an ASCII compatible encoding where ASCII characters are represented by ASCII bytes and non-ASCII characters by sequences of 2-6 non-ASCII bytes, and finally UTF-16 is an extension of UCS-2 in which pairs of certain UCS-2 words can be used to encode non-BMP characters up to 0x10ffff.

To represent wide characters the char type is not suitable. For this reason the ISO C standard introduces a new type that is designed to keep one character of a wide character string. To maintain the similarity there is also a type corresponding to int for those functions that take a single wide character.

— Data type: wchar_t

This data type is used as the base type for wide character strings. In other words, arrays of objects of this type are the equivalent of char[] for multibyte character strings. The type is defined in stddef.h.

The ISO C90 standard, where wchar_t was introduced, does not say anything specific about the representation. It only requires that this type is capable of storing all elements of the basic character set. Therefore it would be legitimate to define wchar_t as char, which might make sense for embedded systems.

But for GNU systems wchar_t is always 32 bits wide and, therefore, capable of representing all UCS-4 values and, therefore, covering all of ISO 10646. Some Unix systems define wchar_t as a 16-bit type and thereby follow Unicode very strictly. This definition is perfectly fine with the standard, but it also means that to represent all characters from Unicode and ISO 10646 one has to use UTF-16 surrogate characters, which is in fact a multi-wide-character encoding. But resorting to multi-wide-character encoding contradicts the purpose of the wchar_t type.

— Data type: wint_t

wint_t is a data type used for parameters and variables that contain a single wide character. As the name suggests this type is the equivalent of int when using the normal char strings. The types wchar_t and wint_t often have the same representation if their size is 32 bits wide but if wchar_t is defined as char the type wint_t must be defined as int due to the parameter promotion.

This type is defined in wchar.h and was introduced in Amendment 1 to ISO C90.

As there are for the char data type macros are available for specifying the minimum and maximum value representable in an object of type wchar_t.

— Macro: wint_t WCHAR_MIN

The macro WCHAR_MIN evaluates to the minimum value representable by an object of type wint_t.

This macro was introduced in Amendment 1 to ISO C90.

— Macro: wint_t WCHAR_MAX

The macro WCHAR_MAX evaluates to the maximum value representable by an object of type wint_t.

This macro was introduced in Amendment 1 to ISO C90.

Another special wide character value is the equivalent to EOF.

— Macro: wint_t WEOF

The macro WEOF evaluates to a constant expression of type wint_t whose value is different from any member of the extended character set.

WEOF need not be the same value as EOF and unlike EOF it also need not be negative. In other words, sloppy code like 

          {
            int c;
            ...
            while ((c = getc (fp)) < 0)
              ...
          }

has to be rewritten to use WEOF explicitly when wide characters are used: 

          {
            wint_t c;
            ...
            while ((c = wgetc (fp)) != WEOF)
              ...
          }

This macro was introduced in Amendment 1 to ISO C90 and is defined in wchar.h.

These internal representations present problems when it comes to storing and transmittal. Because each single wide character consists of more than one byte, they are effected by byte-ordering. Thus, machines with different endianesses would see different values when accessing the same data. This byte ordering concern also applies for communication protocols that are all byte-based and therefore require that the sender has to decide about splitting the wide character in bytes. A last (but not least important) point is that wide characters often require more storage space than a customized byte-oriented character set.

For all the above reasons, an external encoding that is different from the internal encoding is often used if the latter is UCS-2 or UCS-4. The external encoding is byte-based and can be chosen appropriately for the environment and for the texts to be handled. A variety of different character sets can be used for this external encoding (information that will not be exhaustively presented here–instead, a description of the major groups will suffice). All of the ASCII-based character sets fulfill one requirement: they are "filesystem safe." This means that the character '/' is used in the encoding only to represent itself. Things are a bit different for character sets like EBCDIC (Extended Binary Coded Decimal Interchange Code, a character set family used by IBM), but if the operation system does not understand EBCDIC directly the parameters-to-system calls have to be converted first anyhow.

The question remaining is: how to select the character set or encoding to use. The answer: you cannot decide about it yourself, it is decided by the developers of the system or the majority of the users. Since the goal is interoperability one has to use whatever the other people one works with use. If there are no constraints, the selection is based on the requirements the expected circle of users will have. In other words, if a project is expected to be used in only, say, Russia it is fine to use KOI8-R or a similar character set. But if at the same time people from, say, Greece are participating one should use a character set that allows all people to collaborate.

The most widely useful solution seems to be: go with the most general character set, namely ISO 10646. Use UTF-8 as the external encoding and problems about users not being able to use their own language adequately are a thing of the past.

One final comment about the choice of the wide character representation is necessary at this point. We have said above that the natural choice is using Unicode or ISO 10646. This is not required, but at least encouraged, by the ISO C standard. The standard defines at least a macro __STDC_ISO_10646__ that is only defined on systems where the wchar_t type encodes ISO 10646 characters. If this symbol is not defined one should avoid making assumptions about the wide character representation. If the programmer uses only the functions provided by the C library to handle wide character strings there should be no compatibility problems with other systems.

Next: , Previous: Extended Char Intro, Up: Character Set Handling

6.2 Overview about Character Handling Functions

A Unix C library contains three different sets of functions in two families to handle character set conversion. One of the function families (the most commonly used) is specified in the ISO C90 standard and, therefore, is portable even beyond the Unix world. Unfortunately this family is the least useful one. These functions should be avoided whenever possible, especially when developing libraries (as opposed to applications).

The second family of functions got introduced in the early Unix standards (XPG2) and is still part of the latest and greatest Unix standard: Unix 98. It is also the most powerful and useful set of functions. But we will start with the functions defined in Amendment 1 to ISO C90.

Next: , Previous: Charset Function Overview, Up: Character Set Handling

6.3 Restartable Multibyte Conversion Functions

The ISO C standard defines functions to convert strings from a multibyte representation to wide character strings. There are a number of peculiarities:

Despite these limitations the ISO C functions can be used in many contexts. In graphical user interfaces, for instance, it is not uncommon to have functions that require text to be displayed in a wide character string if the text is not simple ASCII. The text itself might come from a file with translations and the user should decide about the current locale, which determines the translation and therefore also the external encoding used. In such a situation (and many others) the functions described here are perfect. If more freedom while performing the conversion is necessary take a look at the iconv functions (see Generic Charset Conversion).

Next: , Up: Restartable multibyte conversion

6.3.1 Selecting the conversion and its properties

We already said above that the currently selected locale for the LC_CTYPE category decides about the conversion that is performed by the functions we are about to describe. Each locale uses its own character set (given as an argument to localedef) and this is the one assumed as the external multibyte encoding. The wide character character set always is UCS-4, at least on GNU systems.

A characteristic of each multibyte character set is the maximum number of bytes that can be necessary to represent one character. This information is quite important when writing code that uses the conversion functions (as shown in the examples below). The ISO C standard defines two macros that provide this information.

— Macro: int MB_LEN_MAX

MB_LEN_MAX specifies the maximum number of bytes in the multibyte sequence for a single character in any of the supported locales. It is a compile-time constant and is defined in limits.h.

— Macro: int MB_CUR_MAX

MB_CUR_MAX expands into a positive integer expression that is the maximum number of bytes in a multibyte character in the current locale. The value is never greater than MB_LEN_MAX. Unlike MB_LEN_MAX this macro need not be a compile-time constant, and in the GNU C library it is not.

MB_CUR_MAX is defined in stdlib.h.

Two different macros are necessary since strictly ISO C90 compilers do not allow variable length array definitions, but still it is desirable to avoid dynamic allocation. This incomplete piece of code shows the problem: 

     {
       char buf[MB_LEN_MAX];
       ssize_t len = 0;
     
       while (! feof (fp))
         {
           fread (&buf[len], 1, MB_CUR_MAX - len, fp);
           /* ... process buf */
           len -= used;
         }
     }

The code in the inner loop is expected to have always enough bytes in the array buf to convert one multibyte character. The array buf has to be sized statically since many compilers do not allow a variable size. The fread call makes sure that MB_CUR_MAX bytes are always available in buf. Note that it isn't a problem if MB_CUR_MAX is not a compile-time constant.

Next: , Previous: Selecting the Conversion, Up: Restartable multibyte conversion

6.3.2 Representing the state of the conversion

In the introduction of this chapter it was said that certain character sets use a stateful encoding. That is, the encoded values depend in some way on the previous bytes in the text.

Since the conversion functions allow converting a text in more than one step we must have a way to pass this information from one call of the functions to another.

— Data type: mbstate_t

A variable of type mbstate_t can contain all the information about the shift state needed from one call to a conversion function to another.

mbstate_t is defined in wchar.h. It was introduced in Amendment 1 to ISO C90.

To use objects of type mbstate_t the programmer has to define such objects (normally as local variables on the stack) and pass a pointer to the object to the conversion functions. This way the conversion function can update the object if the current multibyte character set is stateful.

There is no specific function or initializer to put the state object in any specific state. The rules are that the object should always represent the initial state before the first use, and this is achieved by clearing the whole variable with code such as follows: 

     {
       mbstate_t state;
       memset (&state, '\0', sizeof (state));
       /* from now on state can be used.  */
       ...
     }

When using the conversion functions to generate output it is often necessary to test whether the current state corresponds to the initial state. This is necessary, for example, to decide whether to emit escape sequences to set the state to the initial state at certain sequence points. Communication protocols often require this.

— Function: int mbsinit (const mbstate_t *ps)

The mbsinit function determines whether the state object pointed to by ps is in the initial state. If ps is a null pointer or the object is in the initial state the return value is nonzero. Otherwise it is zero.

mbsinit was introduced in Amendment 1 to ISO C90 and is declared in wchar.h.

Code using mbsinit often looks similar to this: 

     {
       mbstate_t state;
       memset (&state, '\0', sizeof (state));
       /* Use state.  */
       ...
       if (! mbsinit (&state))
         {
           /* Emit code to return to initial state.  */
           const wchar_t empty[] = L"";
           const wchar_t *srcp = empty;
           wcsrtombs (outbuf, &srcp, outbuflen, &state);
         }
       ...
     }

The code to emit the escape sequence to get back to the initial state is interesting. The wcsrtombs function can be used to determine the necessary output code (see Converting Strings). Please note that on GNU systems it is not necessary to perform this extra action for the conversion from multibyte text to wide character text since the wide character encoding is not stateful. But there is nothing mentioned in any standard that prohibits making wchar_t using a stateful encoding.

Next: , Previous: Keeping the state, Up: Restartable multibyte conversion

6.3.3 Converting Single Characters

The most fundamental of the conversion functions are those dealing with single characters. Please note that this does not always mean single bytes. But since there is very often a subset of the multibyte character set that consists of single byte sequences, there are functions to help with converting bytes. Frequently, ASCII is a subpart of the multibyte character set. In such a scenario, each ASCII character stands for itself, and all other characters have at least a first byte that is beyond the range 0 to 127.

— Function: wint_t btowc (int c)

The btowc function (“byte to wide character”) converts a valid single byte character c in the initial shift state into the wide character equivalent using the conversion rules from the currently selected locale of the LC_CTYPE category.

If (unsigned char) c is no valid single byte multibyte character or if c is EOF, the function returns WEOF.

Please note the restriction of c being tested for validity only in the initial shift state. No mbstate_t object is used from which the state information is taken, and the function also does not use any static state.

The btowc function was introduced in Amendment 1 to ISO C90 and is declared in wchar.h.

Despite the limitation that the single byte value always is interpreted in the initial state this function is actually useful most of the time. Most characters are either entirely single-byte character sets or they are extension to ASCII. But then it is possible to write code like this (not that this specific example is very useful): 

     wchar_t *
     itow (unsigned long int val)
     {
       static wchar_t buf[30];
       wchar_t *wcp = &buf[29];
       *wcp = L'\0';
       while (val != 0)
         {
           *--wcp = btowc ('0' + val % 10);
           val /= 10;
         }
       if (wcp == &buf[29])
         *--wcp = L'0';
       return wcp;
     }

Why is it necessary to use such a complicated implementation and not simply cast '0' + val % 10 to a wide character? The answer is that there is no guarantee that one can perform this kind of arithmetic on the character of the character set used for wchar_t representation. In other situations the bytes are not constant at compile time and so the compiler cannot do the work. In situations like this it is necessary btowc.

There also is a function for the conversion in the other direction.

— Function: int wctob (wint_t c)

The wctob function (“wide character to byte”) takes as the parameter a valid wide character. If the multibyte representation for this character in the initial state is exactly one byte long, the return value of this function is this character. Otherwise the return value is EOF.

wctob was introduced in Amendment 1 to ISO C90 and is declared in wchar.h.

There are more general functions to convert single character from multibyte representation to wide characters and vice versa. These functions pose no limit on the length of the multibyte representation and they also do not require it to be in the initial state.

— Function: size_t mbrtowc (wchar_t *restrict pwc, const char *restrict s, size_t n, mbstate_t *restrict ps)

The mbrtowc function (“multibyte restartable to wide character”) converts the next multibyte character in the string pointed to by s into a wide character and stores it in the wide character string pointed to by pwc. The conversion is performed according to the locale currently selected for the LC_CTYPE category. If the conversion for the character set used in the locale requires a state, the multibyte string is interpreted in the state represented by the object pointed to by ps. If ps is a null pointer, a static, internal state variable used only by the mbrtowc function is used.

If the next multibyte character corresponds to the NUL wide character, the return value of the function is 0 and the state object is afterwards in the initial state. If the next n or fewer bytes form a correct multibyte character, the return value is the number of bytes starting from s that form the multibyte character. The conversion state is updated according to the bytes consumed in the conversion. In both cases the wide character (either the L'\0' or the one found in the conversion) is stored in the string pointed to by pwc if pwc is not null.

If the first n bytes of the multibyte string possibly form a valid multibyte character but there are more than n bytes needed to complete it, the return value of the function is (size_t) -2 and no value is stored. Please note that this can happen even if n has a value greater than or equal to MB_CUR_MAX since the input might contain redundant shift sequences.

If the first n bytes of the multibyte string cannot possibly form a valid multibyte character, no value is stored, the global variable errno is set to the value EILSEQ, and the function returns (size_t) -1. The conversion state is afterwards undefined.

mbrtowc was introduced in Amendment 1 to ISO C90 and is declared in wchar.h.

Use of mbrtowc is straightforward. A function that copies a multibyte string into a wide character string while at the same time converting all lowercase characters into uppercase could look like this (this is not the final version, just an example; it has no error checking, and sometimes leaks memory): 

     wchar_t *
     mbstouwcs (const char *s)
     {
       size_t len = strlen (s);
       wchar_t *result = malloc ((len + 1) * sizeof (wchar_t));
       wchar_t *wcp = result;
       wchar_t tmp[1];
       mbstate_t state;
       size_t nbytes;
     
       memset (&state, '\0', sizeof (state));
       while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0)
         {
           if (nbytes >= (size_t) -2)
             /* Invalid input string.  */
             return NULL;
           *wcp++ = towupper (tmp[0]);
           len -= nbytes;
           s += nbytes;
         }
       return result;
     }

The use of mbrtowc should be clear. A single wide character is stored in tmp[0], and the number of consumed bytes is stored in the variable nbytes. If the conversion is successful, the uppercase variant of the wide character is stored in the result array and the pointer to the input string and the number of available bytes is adjusted.

The only non-obvious thing about mbrtowc might be the way memory is allocated for the result. The above code uses the fact that there can never be more wide characters in the converted results than there are bytes in the multibyte input string. This method yields a pessimistic guess about the size of the result, and if many wide character strings have to be constructed this way or if the strings are long, the extra memory required to be allocated because the input string contains multibyte characters might be significant. The allocated memory block can be resized to the correct size before returning it, but a better solution might be to allocate just the right amount of space for the result right away. Unfortunately there is no function to compute the length of the wide character string directly from the multibyte string. There is, however, a function that does part of the work.

— Function: size_t mbrlen (const char *restrict s, size_t n, mbstate_t *ps)

The mbrlen function (“multibyte restartable length”) computes the number of at most n bytes starting at s, which form the next valid and complete multibyte character.

If the next multibyte character corresponds to the NUL wide character, the return value is 0. If the next n bytes form a valid multibyte character, the number of bytes belonging to this multibyte character byte sequence is returned.

If the first n bytes possibly form a valid multibyte character but the character is incomplete, the return value is (size_t) -2. Otherwise the multibyte character sequence is invalid and the return value is (size_t) -1.

The multibyte sequence is interpreted in the state represented by the object pointed to by ps. If ps is a null pointer, a state object local to mbrlen is used.

mbrlen was introduced in Amendment 1 to ISO C90 and is declared in wchar.h.

The attentive reader now will note that mbrlen can be implemented as 

     mbrtowc (NULL, s, n, ps != NULL ? ps : &internal)

This is true and in fact is mentioned in the official specification. How can this function be used to determine the length of the wide character string created from a multibyte character string? It is not directly usable, but we can define a function mbslen using it: 

     size_t
     mbslen (const char *s)
     {
       mbstate_t state;
       size_t result = 0;
       size_t nbytes;
       memset (&state, '\0', sizeof (state));
       while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0)
         {
           if (nbytes >= (size_t) -2)
             /* Something is wrong.  */
             return (size_t) -1;
           s += nbytes;
           ++result;
         }
       return result;
     }

This function simply calls mbrlen for each multibyte character in the string and counts the number of function calls. Please note that we here use MB_LEN_MAX as the size argument in the mbrlen call. This is acceptable since a) this value is larger then the length of the longest multibyte character sequence and b) we know that the string s ends with a NUL byte, which cannot be part of any other multibyte character sequence but the one representing the NUL wide character. Therefore, the mbrlen function will never read invalid memory.

Now that this function is available (just to make this clear, this function is not part of the GNU C library) we can compute the number of wide character required to store the converted multibyte character string s using 

     wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t);

Please note that the mbslen function is quite inefficient. The implementation of mbstouwcs with mbslen would have to perform the conversion of the multibyte character input string twice, and this conversion might be quite expensive. So it is necessary to think about the consequences of using the easier but imprecise method before doing the work twice.

— Function: size_t wcrtomb (char *restrict s, wchar_t wc, mbstate_t *restrict ps)

The wcrtomb function (“wide character restartable to multibyte”) converts a single wide character into a multibyte string corresponding to that wide character.

If s is a null pointer, the function resets the state stored in the objects pointed to by ps (or the internal mbstate_t object) to the initial state. This can also be achieved by a call like this: 

          wcrtombs (temp_buf, L'\0', ps)

since, if s is a null pointer, wcrtomb performs as if it writes into an internal buffer, which is guaranteed to be large enough.

If wc is the NUL wide character, wcrtomb emits, if necessary, a shift sequence to get the state ps into the initial state followed by a single NUL byte, which is stored in the string s.

Otherwise a byte sequence (possibly including shift sequences) is written into the string s. This only happens if wc is a valid wide character (i.e., it has a multibyte representation in the character set selected by locale of the LC_CTYPE category). If wc is no valid wide character, nothing is stored in the strings s, errno is set to EILSEQ, the conversion state in ps is undefined and the return value is (size_t) -1.

If no error occurred the function returns the number of bytes stored in the string s. This includes all bytes representing shift sequences.

One word about the interface of the function: there is no parameter specifying the length of the array s. Instead the function assumes that there are at least MB_CUR_MAX bytes available since this is the maximum length of any byte sequence representing a single character. So the caller has to make sure that there is enough space available, otherwise buffer overruns can occur.

wcrtomb was introduced in Amendment 1 to ISO C90 and is declared in wchar.h.

Using wcrtomb is as easy as using mbrtowc. The following example appends a wide character string to a multibyte character string. Again, the code is not really useful (or correct), it is simply here to demonstrate the use and some problems. 

     char *
     mbscatwcs (char *s, size_t len, const wchar_t *ws)
     {
       mbstate_t state;
       /* Find the end of the existing string.  */
       char *wp = strchr (s, '\0');
       len -= wp - s;
       memset (&state, '\0', sizeof (state));
       do
         {
           size_t nbytes;
           if (len < MB_CUR_LEN)
             {
               /* We cannot guarantee that the next
                  character fits into the buffer, so
                  return an error.  */
               errno = E2BIG;
               return NULL;
             }
           nbytes = wcrtomb (wp, *ws, &state);
           if (nbytes == (size_t) -1)
             /* Error in the conversion.  */
             return NULL;
           len -= nbytes;
           wp += nbytes;
         }
       while (*ws++ != L'\0');
       return s;
     }

First the function has to find the end of the string currently in the array s. The strchr call does this very efficiently since a requirement for multibyte character representations is that the NUL byte is never used except to represent itself (and in this context, the end of the string).

After initializing the state object the loop is entered where the first task is to make sure there is enough room in the array s. We abort if there are not at least MB_CUR_LEN bytes available. This is not always optimal but we have no other choice. We might have less than MB_CUR_LEN bytes available but the next multibyte character might also be only one byte long. At the time the wcrtomb call returns it is too late to decide whether the buffer was large enough. If this solution is unsuitable, there is a very slow but more accurate solution. 

       ...
       if (len < MB_CUR_LEN)
         {
           mbstate_t temp_state;
           memcpy (&temp_state, &state, sizeof (state));
           if (wcrtomb (NULL, *ws, &temp_state) > len)
             {
               /* We cannot guarantee that the next
                  character fits into the buffer, so
                  return an error.  */
               errno = E2BIG;
               return NULL;
             }
         }
       ...

Here we perform the conversion that might overflow the buffer so that we are afterwards in the position to make an exact decision about the buffer size. Please note the NULL argument for the destination buffer in the new wcrtomb call; since we are not interested in the converted text at this point, this is a nice way to express this. The most unusual thing about this piece of code certainly is the duplication of the conversion state object, but if a change of the state is necessary to emit the next multibyte character, we want to have the same shift state change performed in the real conversion. Therefore, we have to preserve the initial shift state information.

There are certainly many more and even better solutions to this problem. This example is only provided for educational purposes.

Next: , Previous: Converting a Character, Up: Restartable multibyte conversion

6.3.4 Converting Multibyte and Wide Character Strings

The functions described in the previous section only convert a single character at a time. Most operations to be performed in real-world programs include strings and therefore the ISO C standard also defines conversions on entire strings. However, the defined set of functions is quite limited; therefore, the GNU C library contains a few extensions that can help in some important situations.

— Function: size_t mbsrtowcs (wchar_t *restrict dst, const char **restrict src, size_t len, mbstate_t *restrict ps)

The mbsrtowcs function (“multibyte string restartable to wide character string”) converts an NUL-terminated multibyte character string at *src into an equivalent wide character string, including the NUL wide character at the end. The conversion is started using the state information from the object pointed to by ps or from an internal object of mbsrtowcs if ps is a null pointer. Before returning, the state object is updated to match the state after the last converted character. The state is the initial state if the terminating NUL byte is reached and converted.

If dst is not a null pointer, the result is stored in the array pointed to by dst; otherwise, the conversion result is not available since it is stored in an internal buffer.

If len wide characters are stored in the array dst before reaching the end of the input string, the conversion stops and len is returned. If dst is a null pointer, len is never checked.

Another reason for a premature return from the function call is if the input string contains an invalid multibyte sequence. In this case the global variable errno is set to EILSEQ and the function returns (size_t) -1.

In all other cases the function returns the number of wide characters converted during this call. If dst is not null, mbsrtowcs stores in the pointer pointed to by src either a null pointer (if the NUL byte in the input string was reached) or the address of the byte following the last converted multibyte character.

mbsrtowcs was introduced in Amendment 1 to ISO C90 and is declared in wchar.h.

The definition of the mbsrtowcs function has one important limitation. The requirement that dst has to be a NUL-terminated string provides problems if one wants to convert buffers with text. A buffer is normally no collection of NUL-terminated strings but instead a continuous collection of lines, separated by newline characters. Now assume that a function to convert one line from a buffer is needed. Since the line is not NUL-terminated, the source pointer cannot directly point into the unmodified text buffer. This means, either one inserts the NUL byte at the appropriate place for the time of the mbsrtowcs function call (which is not doable for a read-only buffer or in a multi-threaded application) or one copies the line in an extra buffer where it can be terminated by a NUL byte. Note that it is not in general possible to limit the number of characters to convert by setting the parameter len to any specific value. Since it is not known how many bytes each multibyte character sequence is in length, one can only guess.

There is still a problem with the method of NUL-terminating a line right after the newline character, which could lead to very strange results. As said in the description of the mbsrtowcs function above the conversion state is guaranteed to be in the initial shift state after processing the NUL byte at the end of the input string. But this NUL byte is not really part of the text (i.e., the conversion state after the newline in the original text could be something different than the initial shift state and therefore the first character of the next line is encoded using this state). But the state in question is never accessible to the user since the conversion stops after the NUL byte (which resets the state). Most stateful character sets in use today require that the shift state after a newline be the initial state–but this is not a strict guarantee. Therefore, simply NUL-terminating a piece of a running text is not always an adequate solution and, therefore, should never be used in generally used code.

The generic conversion interface (see Generic Charset Conversion) does not have this limitation (it simply works on buffers, not strings), and the GNU C library contains a set of functions that take additional parameters specifying the maximal number of bytes that are consumed from the input string. This way the problem of mbsrtowcs's example above could be solved by determining the line length and passing this length to the function.

— Function: size_t wcsrtombs (char *restrict dst, const wchar_t **restrict src, size_t len, mbstate_t *restrict ps)

The wcsrtombs function (“wide character string restartable to multibyte string”) converts the NUL-terminated wide character string at *src into an equivalent multibyte character string and stores the result in the array pointed to by dst. The NUL wide character is also converted. The conversion starts in the state described in the object pointed to by ps or by a state object locally to wcsrtombs in case ps is a null pointer. If dst is a null pointer, the conversion is performed as usual but the result is not available. If all characters of the input string were successfully converted and if dst is not a null pointer, the pointer pointed to by src gets assigned a null pointer.

If one of the wide characters in the input string has no valid multibyte character equivalent, the conversion stops early, sets the global variable errno to EILSEQ, and returns (size_t) -1.

Another reason for a premature stop is if dst is not a null pointer and the next converted character would require more than len bytes in total to the array dst. In this case (and if dest is not a null pointer) the pointer pointed to by src is assigned a value pointing to the wide character right after the last one successfully converted.

Except in the case of an encoding error the return value of the wcsrtombs function is the number of bytes in all the multibyte character sequences stored in dst. Before returning the state in the object pointed to by ps (or the internal object in case ps is a null pointer) is updated to reflect the state after the last conversion. The state is the initial shift state in case the terminating NUL wide character was converted.

The wcsrtombs function was introduced in Amendment 1 to ISO C90 and is declared in wchar.h.

The restriction mentioned above for the mbsrtowcs function applies here also. There is no possibility of directly controlling the number of input characters. One has to place the NUL wide character at the correct place or control the consumed input indirectly via the available output array size (the len parameter).

— Function: size_t mbsnrtowcs (wchar_t *restrict dst, const char **restrict src, size_t nmc, size_t len, mbstate_t *restrict ps)

The mbsnrtowcs function is very similar to the mbsrtowcs function. All the parameters are the same except for nmc, which is new. The return value is the same as for mbsrtowcs.

This new parameter specifies how many bytes at most can be used from the multibyte character string. In other words, the multibyte character string *src need not be NUL-terminated. But if a NUL byte is found within the nmc first bytes of the string, the conversion stops here.

This function is a GNU extension. It is meant to work around the problems mentioned above. Now it is possible to convert a buffer with multibyte character text piece for piece without having to care about inserting NUL bytes and the effect of NUL bytes on the conversion state.

A function to convert a multibyte string into a wide character string and display it could be written like this (this is not a really useful example): 

     void
     showmbs (const char *src, FILE *fp)
     {
       mbstate_t state;
       int cnt = 0;
       memset (&state, '\0', sizeof (state));
       while (1)
         {
           wchar_t linebuf[100];
           const char *endp = strchr (src, '\n');
           size_t n;
     
           /* Exit if there is no more line.  */
           if (endp == NULL)
             break;
     
           n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state);
           linebuf[n] = L'\0';
           fprintf (fp, "line %d: \"%S\"\n", linebuf);
         }
     }

There is no problem with the state after a call to mbsnrtowcs. Since we don't insert characters in the strings that were not in there right from the beginning and we use state only for the conversion of the given buffer, there is no problem with altering the state.

— Function: size_t wcsnrtombs (char *restrict dst, const wchar_t **restrict src, size_t nwc, size_t len, mbstate_t *restrict ps)

The wcsnrtombs function implements the conversion from wide character strings to multibyte character strings. It is similar to wcsrtombs but, just like mbsnrtowcs, it takes an extra parameter, which specifies the length of the input string.

No more than nwc wide characters from the input string *src are converted. If the input string contains a NUL wide character in the first nwc characters, the conversion stops at this place.

The wcsnrtombs function is a GNU extension and just like mbsnrtowcs helps in situations where no NUL-terminated input strings are available.

Previous: Converting Strings, Up: Restartable multibyte conversion

6.3.5 A Complete Multibyte Conversion Example

The example programs given in the last sections are only brief and do not contain all the error checking, etc. Presented here is a complete and documented example. It features the mbrtowc function but it should be easy to derive versions using the other functions. 

     int
     file_mbsrtowcs (int input, int output)
     {
       /* Note the use of MB_LEN_MAX.
          MB_CUR_MAX cannot portably be used here.  */
       char buffer[BUFSIZ + MB_LEN_MAX];
       mbstate_t state;
       int filled = 0;
       int eof = 0;
     
       /* Initialize the state.  */
       memset (&state, '\0', sizeof (state));
     
       while (!eof)
         {
           ssize_t nread;
           ssize_t nwrite;
           char *inp = buffer;
           wchar_t outbuf[BUFSIZ];
           wchar_t *outp = outbuf;
     
           /* Fill up the buffer from the input file.  */
           nread = read (input, buffer + filled, BUFSIZ);
           if (nread < 0)
             {
               perror ("read");
               return 0;
             }
           /* If we reach end of file, make a note to read no more. */
           if (nread == 0)
             eof = 1;
     
           /* filled is now the number of bytes in buffer. */
           filled += nread;
     
           /* Convert those bytes to wide characters--as many as we can. */
           while (1)
             {
               size_t thislen = mbrtowc (outp, inp, filled, &state);
               /* Stop converting at invalid character;
                  this can mean we have read just the first part
                  of a valid character.  */
               if (thislen == (size_t) -1)
                 break;
               /* We want to handle embedded NUL bytes
                  but the return value is 0.  Correct this.  */
               if (thislen == 0)
                 thislen = 1;
               /* Advance past this character. */
               inp += thislen;
               filled -= thislen;
               ++outp;
             }
     
           /* Write the wide characters we just made.  */
           nwrite = write (output, outbuf,
                           (outp - outbuf) * sizeof (wchar_t));
           if (nwrite < 0)
             {
               perror ("write");
               return 0;
             }
     
           /* See if we have a real invalid character. */
           if ((eof && filled > 0) || filled >= MB_CUR_MAX)
             {
               error (0, 0, "invalid multibyte character");
               return 0;
             }
     
           /* If any characters must be carried forward,
              put them at the beginning of buffer. */
           if (filled > 0)
             memmove (inp, buffer, filled);
         }
     
       return 1;
     }

Next: , Previous: Restartable multibyte conversion, Up: Character Set Handling

6.4 Non-reentrant Conversion Function

The functions described in the previous chapter are defined in Amendment 1 to ISO C90, but the original ISO C90 standard also contained functions for character set conversion. The reason that these original functions are not described first is that they are almost entirely useless.

The problem is that all the conversion functions described in the original ISO C90 use a local state. Using a local state implies that multiple conversions at the same time (not only when using threads) cannot be done, and that you cannot first convert single characters and then strings since you cannot tell the conversion functions which state to use.

These original functions are therefore usable only in a very limited set of situations. One must complete converting the entire string before starting a new one, and each string/text must be converted with the same function (there is no problem with the library itself; it is guaranteed that no library function changes the state of any of these functions). For the above reasons it is highly requested that the functions described in the previous section be used in place of non-reentrant conversion functions.

Next: , Up: Non-reentrant Conversion

6.4.1 Non-reentrant Conversion of Single Characters

— Function: int mbtowc (wchar_t *restrict result, const char *restrict string, size_t size)

The mbtowc (“multibyte to wide character”) function when called with non-null string converts the first multibyte character beginning at string to its corresponding wide character code. It stores the result in *result.

mbtowc never examines more than size bytes. (The idea is to supply for size the number of bytes of data you have in hand.)

mbtowc with non-null string distinguishes three possibilities: the first size bytes at string start with valid multibyte characters, they start with an invalid byte sequence or just part of a character, or string points to an empty string (a null character).

For a valid multibyte character, mbtowc converts it to a wide character and stores that in *result, and returns the number of bytes in that character (always at least 1 and never more than size).

For an invalid byte sequence, mbtowc returns -1. For an empty string, it returns 0, also storing '\0' in *result.

If the multibyte character code uses shift characters, then mbtowc maintains and updates a shift state as it scans. If you call mbtowc with a null pointer for string, that initializes the shift state to its standard initial value. It also returns nonzero if the multibyte character code in use actually has a shift state. See Shift State.

— Function: int wctomb (char *string, wchar_t wchar)

The wctomb (“wide character to multibyte”) function converts the wide character code wchar to its corresponding multibyte character sequence, and stores the result in bytes starting at string. At most MB_CUR_MAX characters are stored.

wctomb with non-null string distinguishes three possibilities for wchar: a valid wide character code (one that can be translated to a multibyte character), an invalid code, and L'\0'.

Given a valid code, wctomb converts it to a multibyte character, storing the bytes starting at string. Then it returns the number of bytes in that character (always at least 1 and never more than MB_CUR_MAX).

If wchar is an invalid wide character code, wctomb returns -1. If wchar is L'\0', it returns 0, also storing '\0' in *string.

If the multibyte character code uses shift characters, then wctomb maintains and updates a shift state as it scans. If you call wctomb with a null pointer for string, that initializes the shift state to its standard initial value. It also returns nonzero if the multibyte character code in use actually has a shift state. See Shift State.

Calling this function with a wchar argument of zero when string is not null has the side-effect of reinitializing the stored shift state as well as storing the multibyte character '\0' and returning 0.

Similar to mbrlen there is also a non-reentrant function that computes the length of a multibyte character. It can be defined in terms of mbtowc.

— Function: int mblen (const char *string, size_t size)

The mblen function with a non-null string argument returns the number of bytes that make up the multibyte character beginning at string, never examining more than size bytes. (The idea is to supply for size the number of bytes of data you have in hand.)

The return value of mblen distinguishes three possibilities: the first size bytes at string start with valid multibyte characters, they start with an invalid byte sequence or just part of a character, or string points to an empty string (a null character).

For a valid multibyte character, mblen returns the number of bytes in that character (always at least 1 and never more than size). For an invalid byte sequence, mblen returns -1. For an empty string, it returns 0.

If the multibyte character code uses shift characters, then mblen maintains and updates a shift state as it scans. If you call mblen with a null pointer for string, that initializes the shift state to its standard initial value. It also returns a nonzero value if the multibyte character code in use actually has a shift state. See Shift State.

The function mblen is declared in stdlib.h.

Next: , Previous: Non-reentrant Character Conversion, Up: Non-reentrant Conversion

6.4.2 Non-reentrant Conversion of Strings

For convenience the ISO C90 standard also defines functions to convert entire strings instead of single characters. These functions suffer from the same problems as their reentrant counterparts from Amendment 1 to ISO C90; see Converting Strings.

— Function: size_t mbstowcs (wchar_t *wstring, const char *string, size_t size)

The mbstowcs (“multibyte string to wide character string”) function converts the null-terminated string of multibyte characters string to an array of wide character codes, storing not more than size wide characters into the array beginning at wstring. The terminating null character counts towards the size, so if size is less than the actual number of wide characters resulting from string, no terminating null character is stored.

The conversion of characters from string begins in the initial shift state.

If an invalid multibyte character sequence is found, the mbstowcs function returns a value of -1. Otherwise, it returns the number of wide characters stored in the array wstring. This number does not include the terminating null character, which is present if the number is less than size.

Here is an example showing how to convert a string of multibyte characters, allocating enough space for the result. 

          wchar_t *
          mbstowcs_alloc (const char *string)
          {
            size_t size = strlen (string) + 1;
            wchar_t *buf = xmalloc (size * sizeof (wchar_t));
          
            size = mbstowcs (buf, string, size);
            if (size == (size_t) -1)
              return NULL;
            buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));
            return buf;
          }
— Function: size_t wcstombs (char *string, const wchar_t *wstring, size_t size)

The wcstombs (“wide character string to multibyte string”) function converts the null-terminated wide character array wstring into a string containing multibyte characters, storing not more than size bytes starting at string, followed by a terminating null character if there is room. The conversion of characters begins in the initial shift state.

The terminating null character counts towards the size, so if size is less than or equal to the number of bytes needed in wstring, no terminating null character is stored.

If a code that does not correspond to a valid multibyte character is found, the wcstombs function returns a value of -1. Otherwise, the return value is the number of bytes stored in the array string. This number does not include the terminating null character, which is present if the number is less than size.

Previous: Non-reentrant String Conversion, Up: Non-reentrant Conversion

6.4.3 States in Non-reentrant Functions

In some multibyte character codes, the meaning of any particular byte sequence is not fixed; it depends on what other sequences have come earlier in the same string. Typically there are just a few sequences that can change the meaning of other sequences; these few are called shift sequences and we say that they set the shift state for other sequences that follow.

To illustrate shift state and shift sequences, suppose we decide that the sequence 0200 (just one byte) enters Japanese mode, in which pairs of bytes in the range from 0240 to 0377 are single characters, while 0201 enters Latin-1 mode, in which single bytes in the range from 0240 to 0377 are characters, and interpreted according to the ISO Latin-1 character set. This is a multibyte code that has two alternative shift states (“Japanese mode” and “Latin-1 mode”), and two shift sequences that specify particular shift states.

When the multibyte character code in use has shift states, then mblen, mbtowc, and wctomb must maintain and update the current shift state as they scan the string. To make this work properly, you must follow these rules:

Here is an example of using mblen following these rules: 

     void
     scan_string (char *s)
     {
       int length = strlen (s);
     
       /* Initialize shift state.  */
       mblen (NULL, 0);
     
       while (1)
         {
           int thischar = mblen (s, length);
           /* Deal with end of string and invalid characters.  */
           if (thischar == 0)
             break;
           if (thischar == -1)
             {
               error ("invalid multibyte character");
               break;
             }
           /* Advance past this character.  */
           s += thischar;
           length -= thischar;
         }
     }

The functions mblen, mbtowc and wctomb are not reentrant when using a multibyte code that uses a shift state. However, no other library functions call these functions, so you don't have to worry that the shift state will be changed mysteriously.

Previous: Non-reentrant Conversion, Up: Character Set Handling

6.5 Generic Charset Conversion

The conversion functions mentioned so far in this chapter all had in common that they operate on character sets that are not directly specified by the functions. The multibyte encoding used is specified by the currently selected locale for the LC_CTYPE category. The wide character set is fixed by the implementation (in the case of GNU C library it is always UCS-4 encoded ISO 10646.

This has of course several problems when it comes to general character conversion:

The XPG2 standard defines a completely new set of functions, which has none of these limitations. They are not at all coupled to the selected locales, and they have no constraints on the character sets selected for source and destination. Only the set of available conversions limits them. The standard does not specify that any conversion at all must be available. Such availability is a measure of the quality of the implementation.

In the following text first the interface to iconv and then the conversion function, will be described. Comparisons with other implementations will show what obstacles stand in the way of portable applications. Finally, the implementation is described in so far as might interest the advanced user who wants to extend conversion capabilities.

Next: , Up: Generic Charset Conversion

6.5.1 Generic Character Set Conversion Interface

This set of functions follows the traditional cycle of using a resource: open–use–close. The interface consists of three functions, each of which implements one step.

Before the interfaces are described it is necessary to introduce a data type. Just like other open–use–close interfaces the functions introduced here work using handles and the iconv.h header defines a special type for the handles used.

— Data Type: iconv_t

This data type is an abstract type defined in iconv.h. The user must not assume anything about the definition of this type; it must be completely opaque.

Objects of this type can get assigned handles for the conversions using the iconv functions. The objects themselves need not be freed, but the conversions for which the handles stand for have to.

The first step is the function to create a handle.

— Function: iconv_t iconv_open (const char *tocode, const char *fromcode)

The iconv_open function has to be used before starting a conversion. The two parameters this function takes determine the source and destination character set for the conversion, and if the implementation has the possibility to perform such a conversion, the function returns a handle.

If the wanted conversion is not available, the iconv_open function returns (iconv_t) -1. In this case the global variable errno can have the following values:

EMFILE
The process already has OPEN_MAX file descriptors open.
ENFILE
The system limit of open file is reached.
ENOMEM
Not enough memory to carry out the operation.
EINVAL
The conversion from fromcode to tocode is not supported.

It is not possible to use the same descriptor in different threads to perform independent conversions. The data structures associated with the descriptor include information about the conversion state. This must not be messed up by using it in different conversions.

An iconv descriptor is like a file descriptor as for every use a new descriptor must be created. The descriptor does not stand for all of the conversions from fromset to toset.

The GNU C library implementation of iconv_open has one significant extension to other implementations. To ease the extension of the set of available conversions, the implementation allows storing the necessary files with data and code in an arbitrary number of directories. How this extension must be written will be explained below (see glibc iconv Implementation). Here it is only important to say that all directories mentioned in the GCONV_PATH environment variable are considered only if they contain a file gconv-modules. These directories need not necessarily be created by the system administrator. In fact, this extension is introduced to help users writing and using their own, new conversions. Of course, this does not work for security reasons in SUID binaries; in this case only the system directory is considered and this normally is prefix/lib/gconv. The GCONV_PATH environment variable is examined exactly once at the first call of the iconv_open function. Later modifications of the variable have no effect.

The iconv_open function was introduced early in the X/Open Portability Guide, version 2. It is supported by all commercial Unices as it is required for the Unix branding. However, the quality and completeness of the implementation varies widely. The iconv_open function is declared in iconv.h.

The iconv implementation can associate large data structure with the handle returned by iconv_open. Therefore, it is crucial to free all the resources once all conversions are carried out and the conversion is not needed anymore.

— Function: int iconv_close (iconv_t cd)

The iconv_close function frees all resources associated with the handle cd, which must have been returned by a successful call to the iconv_open function.

If the function call was successful the return value is 0. Otherwise it is -1 and errno is set appropriately. Defined error are:

EBADF
The conversion descriptor is invalid.

The iconv_close function was introduced together with the rest of the iconv functions in XPG2 and is declared in iconv.h.

The standard defines only one actual conversion function. This has, therefore, the most general interface: it allows conversion from one buffer to another. Conversion from a file to a buffer, vice versa, or even file to file can be implemented on top of it.

— Function: size_t iconv (iconv_t cd, char **inbuf, size_t *inbytesleft, char **outbuf, size_t *outbytesleft)

The iconv function converts the text in the input buffer according to the rules associated with the descriptor cd and stores the result in the output buffer. It is possible to call the function for the same text several times in a row since for stateful character sets the necessary state information is kept in the data structures associated with the descriptor.

The input buffer is specified by *inbuf and it contains *inbytesleft bytes. The extra indirection is necessary for communicating the used input back to the caller (see below). It is important to note that the buffer pointer is of type char and the length is measured in bytes even if the input text is encoded in wide characters.

The output buffer is specified in a similar way. *outbuf points to the beginning of the buffer with at least *outbytesleft bytes room for the result. The buffer pointer again is of type char and the length is measured in bytes. If outbuf or *outbuf is a null pointer, the conversion is performed but no output is available.

If inbuf is a null pointer, the iconv function performs the necessary action to put the state of the conversion into the initial state. This is obviously a no-op for non-stateful encodings, but if the encoding has a state, such a function call might put some byte sequences in the output buffer, which perform the necessary state changes. The next call with inbuf not being a null pointer then simply goes on from the initial state. It is important that the programmer never makes any assumption as to whether the conversion has to deal with states. Even if the input and output character sets are not stateful, the implementation might still have to keep states. This is due to the implementation chosen for the GNU C library as it is described below. Therefore an iconv call to reset the state should always be performed if some protocol requires this for the output text.

The conversion stops for one of three reasons. The first is that all characters from the input buffer are converted. This actually can mean two things: either all bytes from the input buffer are consumed or there are some bytes at the end of the buffer that possibly can form a complete character but the input is incomplete. The second reason for a stop is that the output buffer is full. And the third reason is that the input contains invalid characters.

In all of these cases the buffer pointers after the last successful conversion, for input and output buffer, are stored in inbuf and outbuf, and the available room in each buffer is stored in inbytesleft and outbytesleft.

Since the character sets selected in the iconv_open call can be almost arbitrary, there can be situations where the input buffer contains valid characters, which have no identical representation in the output character set. The behavior in this situation is undefined. The current behavior of the GNU C library in this situation is to return with an error immediately. This certainly is not the most desirable solution; therefore, future versions will provide better ones, but they are not yet finished.

If all input from the input buffer is successfully converted and stored in the output buffer, the function returns the number of non-reversible conversions performed. In all other cases the return value is (size_t) -1 and errno is set appropriately. In such cases the value pointed to by inbytesleft is nonzero.

EILSEQ
The conversion stopped because of an invalid byte sequence in the input. After the call, *inbuf points at the first byte of the invalid byte sequence.
E2BIG
The conversion stopped because it ran out of space in the output buffer.
EINVAL
The conversion stopped because of an incomplete byte sequence at the end of the input buffer.
EBADF
The cd argument is invalid.

The iconv function was introduced in the XPG2 standard and is declared in the iconv.h header.

The definition of the iconv function is quite good overall. It provides quite flexible functionality. The only problems lie in the boundary cases, which are incomplete byte sequences at the end of the input buffer and invalid input. A third problem, which is not really a design problem, is the way conversions are selected. The standard does not say anything about the legitimate names, a minimal set of available conversions. We will see how this negatively impacts other implementations, as demonstrated below.

Next: , Previous: Generic Conversion Interface, Up: Generic Charset Conversion

6.5.2 A complete iconv example

The example below features a solution for a common problem. Given that one knows the internal encoding used by the system for wchar_t strings, one often is in the position to read text from a file and store it in wide character buffers. One can do this using mbsrtowcs, but then we run into the problems discussed above. 

     int
     file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail)
     {
       char inbuf[BUFSIZ];
       size_t insize = 0;
       char *wrptr = (char *) outbuf;
       int result = 0;
       iconv_t cd;
     
       cd = iconv_open ("WCHAR_T", charset);
       if (cd == (iconv_t) -1)
         {
           /* Something went wrong.  */
           if (errno == EINVAL)
             error (0, 0, "conversion from '%s' to wchar_t not available",
                    charset);
           else
             perror ("iconv_open");
     
           /* Terminate the output string.  */
           *outbuf = L'\0';
     
           return -1;
         }
     
       while (avail > 0)
         {
           size_t nread;
           size_t nconv;
           char *inptr = inbuf;
     
           /* Read more input.  */
           nread = read (fd, inbuf + insize, sizeof (inbuf) - insize);
           if (nread == 0)
             {
               /* When we come here the file is completely read.
                  This still could mean there are some unused
                  characters in the inbuf.  Put them back.  */
               if (lseek (fd, -insize, SEEK_CUR) == -1)
                 result = -1;
     
               /* Now write out the byte sequence to get into the
                  initial state if this is necessary.  */
               iconv (cd, NULL, NULL, &wrptr, &avail);
     
               break;
             }
           insize += nread;
     
           /* Do the conversion.  */
           nconv = iconv (cd, &inptr, &insize, &wrptr, &avail);
           if (nconv == (size_t) -1)
             {
               /* Not everything went right.  It might only be
                  an unfinished byte sequence at the end of the
                  buffer.  Or it is a real problem.  */
               if (errno == EINVAL)
                 /* This is harmless.  Simply move the unused
                    bytes to the beginning of the buffer so that
                    they can be used in the next round.  */
                 memmove (inbuf, inptr, insize);
               else
                 {
                   /* It is a real problem.  Maybe we ran out of
                      space in the output buffer or we have invalid
                      input.  In any case back the file pointer to
                      the position of the last processed byte.  */
                   lseek (fd, -insize, SEEK_CUR);
                   result = -1;
                   break;
                 }
             }
         }
     
       /* Terminate the output string.  */
       if (avail >= sizeof (wchar_t))
         *((wchar_t *) wrptr) = L'\0';
     
       if (iconv_close (cd) != 0)
         perror ("iconv_close");
     
       return (wchar_t *) wrptr - outbuf;
     }

This example shows the most important aspects of using the iconv functions. It shows how successive calls to iconv can be used to convert large amounts of text. The user does not have to care about stateful encodings as the functions take care of everything.

An interesting point is the case where iconv returns an error and errno is set to EINVAL. This is not really an error in the transformation. It can happen whenever the input character set contains byte sequences of more than one byte for some character and texts are not processed in one piece. In this case there is a chance that a multibyte sequence is cut. The caller can then simply read the remainder of the takes and feed the offending bytes together with new character from the input to iconv and continue the work. The internal state kept in the descriptor is not unspecified after such an event as is the case with the conversion functions from the ISO C standard.

The example also shows the problem of using wide character strings with iconv. As explained in the description of the iconv function above, the function always takes a pointer to a char array and the available space is measured in bytes. In the example, the output buffer is a wide character buffer; therefore, we use a local variable wrptr of type char *, which is used in the iconv calls.

This looks rather innocent but can lead to problems on platforms that have tight restriction on alignment. Therefore the caller of iconv has to make sure that the pointers passed are suitable for access of characters from the appropriate character set. Since, in the above case, the input parameter to the function is a wchar_t pointer, this is the case (unless the user violates alignment when computing the parameter). But in other situations, especially when writing generic functions where one does not know what type of character set one uses and, therefore, treats text as a sequence of bytes, it might become tricky.

Next: , Previous: iconv Examples, Up: Generic Charset Conversion

6.5.3 Some Details about other iconv Implementations

This is not really the place to discuss the iconv implementation of other systems but it is necessary to know a bit about them to write portable programs. The above mentioned problems with the specification of the iconv functions can lead to portability issues.

The first thing to notice is that, due to the large number of character sets in use, it is certainly not practical to encode the conversions directly in the C library. Therefore, the conversion information must come from files outside the C library. This is usually done in one or both of the following ways:

Some implementations in commercial Unices implement a mixture of these possibilities; the majority implement only the second solution. Using loadable modules moves the code out of the library itself and keeps the door open for extensions and improvements, but this design is also limiting on some platforms since not many platforms support dynamic loading in statically linked programs. On platforms without this capability it is therefore not possible to use this interface in statically linked programs. The GNU C library has, on ELF platforms, no problems with dynamic loading in these situations; therefore, this point is moot. The danger is that one gets acquainted with this situation and forgets about the restrictions on other systems.

A second thing to know about other iconv implementations is that the number of available conversions is often very limited. Some implementations provide, in the standard release (not special international or developer releases), at most 100 to 200 conversion possibilities. This does not mean 200 different character sets are supported; for example, conversions from one character set to a set of 10 others might count as 10 conversions. Together with the other direction this makes 20 conversion possibilities used up by one character set. One can imagine the thin coverage these platform provide. Some Unix vendors even provide only a handful of conversions, which renders them useless for almost all uses.

This directly leads to a third and probably the most problematic point. The way the iconv conversion functions are implemented on all known Unix systems and the availability of the conversion functions from character set A to B and the conversion from B to C does not imply that the conversion from A to C is available.

This might not seem unreasonable and problematic at first, but it is a quite big problem as one will notice shortly after hitting it. To show the problem we assume to write a program that has to convert from A to C. A call like 

     cd = iconv_open ("C", "A");

fails according to the assumption above. But what does the program do now? The conversion is necessary; therefore, simply giving up is not an option.

This is a nuisance. The iconv function should take care of this. But how should the program proceed from here on? If it tries to convert to character set B, first the two iconv_open calls 

     cd1 = iconv_open ("B", "A");

and 

     cd2 = iconv_open ("C", "B");

will succeed, but how to find B?

Unfortunately, the answer is: there is no general solution. On some systems guessing might help. On those systems most character sets can convert to and from UTF-8 encoded ISO 10646 or Unicode text. Beside this only some very system-specific methods can help. Since the conversion functions come from loadable modules and these modules must be stored somewhere in the filesystem, one could try to find them and determine from the available file which conversions are available and whether there is an indirect route from A to C.

This example shows one of the design errors of iconv mentioned above. It should at least be possible to determine the list of available conversion programmatically so that if iconv_open says there is no such conversion, one could make sure this also is true for indirect routes.

Previous: Other iconv Implementations, Up: Generic Charset Conversion

6.5.4 The iconv Implementation in the GNU C library

After reading about the problems of iconv implementations in the last section it is certainly good to note that the implementation in the GNU C library has none of the problems mentioned above. What follows is a step-by-step analysis of the points raised above. The evaluation is based on the current state of the development (as of January 1999). The development of the iconv functions is not complete, but basic functionality has solidified.

The GNU C library's iconv implementation uses shared loadable modules to implement the conversions. A very small number of conversions are built into the library itself but these are only rather trivial conversions.

All the benefits of loadable modules are available in the GNU C library implementation. This is especially appealing since the interface is well documented (see below), and it, therefore, is easy to write new conversion modules. The drawback of using loadable objects is not a problem in the GNU C library, at least on ELF systems. Since the library is able to load shared objects even in statically linked binaries, static linking need not be forbidden in case one wants to use iconv.

The second mentioned problem is the number of supported conversions. Currently, the GNU C library supports more than 150 character sets. The way the implementation is designed the number of supported conversions is greater than 22350 (150 times 149). If any conversion from or to a character set is missing, it can be added easily.

Particularly impressive as it may be, this high number is due to the fact that the GNU C library implementation of iconv does not have the third problem mentioned above (i.e., whenever there is a conversion from a character set A to B and from B to C it is always possible to convert from A to C directly). If the iconv_open returns an error and sets errno to EINVAL, there is no known way, directly or indirectly, to perform the wanted conversion.

Triangulation is achieved by providing for each character set a conversion from and to UCS-4 encoded ISO 10646. Using ISO 10646 as an intermediate representation it is possible to triangulate (i.e., convert with an intermediate representation).

There is no inherent requirement to provide a conversion to ISO 10646 for a new character set, and it is also possible to provide other conversions where neither source nor destination character set is ISO 10646. The existing set of conversions is simply meant to cover all conversions that might be of interest.

All currently available conversions use the triangulation method above, making conversion run unnecessarily slow. If, for example, somebody often needs the conversion from ISO-2022-JP to EUC-JP, a quicker solution would involve direct conversion between the two character sets, skipping the input to ISO 10646 first. The two character sets of interest are much more similar to each other than to ISO 10646.

In such a situation one easily can write a new conversion and provide it as a better alternative. The GNU C library iconv implementation would automatically use the module implementing the conversion if it is specified to be more efficient.

6.5.4.1 Format of gconv-modules files

All information about the available conversions comes from a file named gconv-modules, which can be found in any of the directories along the GCONV_PATH. The gconv-modules files are line-oriented text files, where each of the lines has one of the following formats:

Returning to the example above where one has written a module to directly convert from ISO-2022-JP to EUC-JP and back. All that has to be done is to put the new module, let its name be ISO2022JP-EUCJP.so, in a directory and add a file gconv-modules with the following content in the same directory: 

     module  ISO-2022-JP//   EUC-JP//        ISO2022JP-EUCJP    1
     module  EUC-JP//        ISO-2022-JP//   ISO2022JP-EUCJP    1

To see why this is sufficient, it is necessary to understand how the conversion used by iconv (and described in the descriptor) is selected. The approach to this problem is quite simple.

At the first call of the iconv_open function the program reads all available gconv-modules files and builds up two tables: one containing all the known aliases and another that contains the information about the conversions and which shared object implements them.

6.5.4.2 Finding the conversion path in iconv

The set of available conversions form a directed graph with weighted edges. The weights on the edges are the costs specified in the gconv-modules files. The iconv_open function uses an algorithm suitable for search for the best path in such a graph and so constructs a list of conversions that must be performed in succession to get the transformation from the source to the destination character set.

Explaining why the above gconv-modules files allows the iconv implementation to resolve the specific ISO-2022-JP to EUC-JP conversion module instead of the conversion coming with the library itself is straightforward. Since the latter conversion takes two steps (from ISO-2022-JP to ISO 10646 and then from ISO 10646 to EUC-JP), the cost is 1+1 = 2. The above gconv-modules file, however, specifies that the new conversion modules can perform this conversion with only the cost of 1.

A mysterious item about the gconv-modules file above (and also the file coming with the GNU C library) are the names of the character sets specified in the module lines. Why do almost all the names end in //? And this is not all: the names can actually be regular expressions. At this point in time this mystery should not be revealed, unless you have the relevant spell-casting materials: ashes from an original DOS 6.2 boot disk burnt in effigy, a crucifix blessed by St. Emacs, assorted herbal roots from Central America, sand from Cebu, etc. Sorry! The part of the implementation where this is used is not yet finished. For now please simply follow the existing examples. It'll become clearer once it is. –drepper

A last remark about the gconv-modules is about the names not ending with //. A character set named INTERNAL is often mentioned. From the discussion above and the chosen name it should have become clear that this is the name for the representation used in the intermediate step of the triangulation. We have said that this is UCS-4 but actually that is not quite right. The UCS-4 specification also includes the specification of the byte ordering used. Since a UCS-4 value consists of four bytes, a stored value is effected by byte ordering. The internal representation is not the same as UCS-4 in case the byte ordering of the processor (or at least the running process) is not the same as the one required for UCS-4. This is done for performance reasons as one does not want to perform unnecessary byte-swapping operations if one is not interested in actually seeing the result in UCS-4. To avoid trouble with endianness, the internal representation consistently is named INTERNAL even on big-endian systems where the representations are identical.

6.5.4.3 iconv module data structures

So far this section has described how modules are located and considered to be used. What remains to be described is the interface of the modules so that one can write new ones. This section describes the interface as it is in use in January 1999. The interface will change a bit in the future but, with luck, only in an upwardly compatible way.

The definitions necessary to write new modules are publicly available in the non-standard header gconv.h. The following text, therefore, describes the definitions from this header file. First, however, it is necessary to get an overview.

From the perspective of the user of iconv the interface is quite simple: the iconv_open function returns a handle that can be used in calls to iconv, and finally the handle is freed with a call to iconv_close. The problem is that the handle has to be able to represent the possibly long sequences of conversion steps and also the state of each conversion since the handle is all that is passed to the iconv function. Therefore, the data structures are really the elements necessary to understanding the implementation.

We need two different kinds of data structures. The first describes the conversion and the second describes the state etc. There are really two type definitions like this in gconv.h.

— Data type: struct __gconv_step

This data structure describes one conversion a module can perform. For each function in a loaded module with conversion functions there is exactly one object of this type. This object is shared by all users of the conversion (i.e., this object does not contain any information corresponding to an actual conversion; it only describes the conversion itself).

struct __gconv_loaded_object *__shlib_handle
const char *__modname
int __counter
All these elements of the structure are used internally in the C library to coordinate loading and unloading the shared. One must not expect any of the other elements to be available or initialized.
const char *__from_name
const char *__to_name
__from_name and __to_name contain the names of the source and destination character sets. They can be used to identify the actual conversion to be carried out since one module might implement conversions for more than one character set and/or direction.
gconv_fct __fct
gconv_init_fct __init_fct
gconv_end_fct __end_fct
These elements contain pointers to the functions in the loadable module. The interface will be explained below.
int __min_needed_from
int __max_needed_from
int __min_needed_to
int __max_needed_to;
These values have to be supplied in the init function of the module. The __min_needed_from value specifies how many bytes a character of the source character set at least needs. The __max_needed_from specifies the maximum value that also includes possible shift sequences.

The __min_needed_to and __max_needed_to values serve the same purpose as __min_needed_from and __max_needed_from but this time for the destination character set.

It is crucial that these values be accurate since otherwise the conversion functions will have problems or not work at all.

int __stateful
This element must also be initialized by the init function. int __stateful is nonzero if the source character set is stateful. Otherwise it is zero.
void *__data
This element can be used freely by the conversion functions in the module. void *__data can be used to communicate extra information from one call to another. void *__data need not be initialized if not needed at all. If void *__data element is assigned a pointer to dynamically allocated memory (presumably in the init function) it has to be made sure that the end function deallocates the memory. Otherwise the application will leak memory.

It is important to be aware that this data structure is shared by all users of this specification conversion and therefore the __data element must not contain data specific to one specific use of the conversion function.

— Data type: struct __gconv_step_data

This is the data structure that contains the information specific to each use of the conversion functions.

char *__outbuf
char *__outbufend
These elements specify the output buffer for the conversion step. The __outbuf element points to the beginning of the buffer, and __outbufend points to the byte following the last byte in the buffer. The conversion function must not assume anything about the size of the buffer but it can be safely assumed the there is room for at least one complete character in the output buffer.

Once the conversion is finished, if the conversion is the last step, the __outbuf element must be modified to point after the last byte written into the buffer to signal how much output is available. If this conversion step is not the last one, the element must not be modified. The __outbufend element must not be modified.

int __is_last
This element is nonzero if this conversion step is the last one. This information is necessary for the recursion. See the description of the conversion function internals below. This element must never be modified.
int __invocation_counter
The conversion function can use this element to see how many calls of the conversion function already happened. Some character sets require a certain prolog when generating output, and by comparing this value with zero, one can find out whether it is the first call and whether, therefore, the prolog should be emitted. This element must never be modified.
int __internal_use
This element is another one rarely used but needed in certain situations. It is assigned a nonzero value in case the conversion functions are used to implement mbsrtowcs et.al. (i.e., the function is not used directly through the iconv interface).

This sometimes makes a difference as it is expected that the iconv functions are used to translate entire texts while the mbsrtowcs functions are normally used only to convert single strings and might be used multiple times to convert entire texts.

But in this situation we would have problem complying with some rules of the character set specification. Some character sets require a prolog, which must appear exactly once for an entire text. If a number of mbsrtowcs calls are used to convert the text, only the first call must add the prolog. However, because there is no communication between the different calls of mbsrtowcs, the conversion functions have no possibility to find this out. The situation is different for sequences of iconv calls since the handle allows access to the needed information.

The int __internal_use element is mostly used together with __invocation_counter as follows: 

               if (!data->__internal_use
                    && data->__invocation_counter == 0)
                 /* Emit prolog.  */
                 ...

This element must never be modified.

mbstate_t *__statep
The __statep element points to an object of type mbstate_t (see Keeping the state). The conversion of a stateful character set must use the object pointed to by __statep to store information about the conversion state. The __statep element itself must never be modified.
mbstate_t __state
This element must never be used directly. It is only part of this structure to have the needed space allocated.

6.5.4.4 iconv module interfaces

With the knowledge about the data structures we now can describe the conversion function itself. To understand the interface a bit of knowledge is necessary about the functionality in the C library that loads the objects with the conversions.

It is often the case that one conversion is used more than once (i.e., there are several iconv_open calls for the same set of character sets during one program run). The mbsrtowcs et.al. functions in the GNU C library also use the iconv functionality, which increases the number of uses of the same functions even more.

Because of this multiple use of conversions, the modules do not get loaded exclusively for one conversion. Instead a module once loaded can be used by an arbitrary number of iconv or mbsrtowcs calls at the same time. The splitting of the information between conversion- function-specific information and conversion data makes this possible. The last section showed the two data structures used to do this.

This is of course also reflected in the interface and semantics of the functions that the modules must provide. There are three functions that must have the following names:

gconv_init
The gconv_init function initializes the conversion function specific data structure. This very same object is shared by all conversions that use this conversion and, therefore, no state information about the conversion itself must be stored in here. If a module implements more than one conversion, the gconv_init function will be called multiple times.
gconv_end
The gconv_end function is responsible for freeing all resources allocated by the gconv_init function. If there is nothing to do, this function can be missing. Special care must be taken if the module implements more than one conversion and the gconv_init function does not allocate the same resources for all conversions.
gconv
This is the actual conversion function. It is called to convert one block of text. It gets passed the conversion step information initialized by gconv_init and the conversion data, specific to this use of the conversion functions.

There are three data types defined for the three module interface functions and these define the interface.

— Data type: int (*__gconv_init_fct) (struct __gconv_step *)

This specifies the interface of the initialization function of the module. It is called exactly once for each conversion the module implements.

As explained in the description of the struct __gconv_step data structure above the initialization function has to initialize parts of it.

__min_needed_from
__max_needed_from
__min_needed_to
__max_needed_to
These elements must be initialized to the exact numbers of the minimum and maximum number of bytes used by one character in the source and destination character sets, respectively. If the characters all have the same size, the minimum and maximum values are the same.
__stateful
This element must be initialized to an nonzero value if the source character set is stateful. Otherwise it must be zero.

If the initialization function needs to communicate some information to the conversion function, this communication can happen using the __data element of the __gconv_step structure. But since this data is shared by all the conversions, it must not be modified by the conversion function. The example below shows how this can be used. 

          #define MIN_NEEDED_FROM         1
          #define MAX_NEEDED_FROM         4
          #define MIN_NEEDED_TO           4
          #define MAX_NEEDED_TO           4
          
          int
          gconv_init (struct __gconv_step *step)
          {
            /* Determine which direction.  */
            struct iso2022jp_data *new_data;
            enum direction dir = illegal_dir;
            enum variant var = illegal_var;
            int result;
          
            if (__strcasecmp (step->__from_name, "ISO-2022-JP//") == 0)
              {
                dir = from_iso2022jp;
                var = iso2022jp;
              }
            else if (__strcasecmp (step->__to_name, "ISO-2022-JP//") == 0)
              {
                dir = to_iso2022jp;
                var = iso2022jp;
              }
            else if (__strcasecmp (step->__from_name, "ISO-2022-JP-2//") == 0)
              {
                dir = from_iso2022jp;
                var = iso2022jp2;
              }
            else if (__strcasecmp (step->__to_name, "ISO-2022-JP-2//") == 0)
              {
                dir = to_iso2022jp;
                var = iso2022jp2;
              }
          
            result = __GCONV_NOCONV;
            if (dir != illegal_dir)
              {
                new_data = (struct iso2022jp_data *)
                  malloc (sizeof (struct iso2022jp_data));
          
                result = __GCONV_NOMEM;
                if (new_data != NULL)
                  {
                    new_data->dir = dir;
                    new_data->var = var;
                    step->__data = new_data;
          
                    if (dir == from_iso2022jp)
                      {
                        step->__min_needed_from = MIN_NEEDED_FROM;
                        step->__max_needed_from = MAX_NEEDED_FROM;
                        step->__min_needed_to = MIN_NEEDED_TO;
                        step->__max_needed_to = MAX_NEEDED_TO;
                      }
                    else
                      {
                        step->__min_needed_from = MIN_NEEDED_TO;
                        step->__max_needed_from = MAX_NEEDED_TO;
                        step->__min_needed_to = MIN_NEEDED_FROM;
                        step->__max_needed_to = MAX_NEEDED_FROM + 2;
                      }
          
                    /* Yes, this is a stateful encoding.  */
                    step->__stateful = 1;
          
                    result = __GCONV_OK;
                  }
              }
          
            return result;
          }

The function first checks which conversion is wanted. The module from which this function is taken implements four different conversions; which one is selected can be determined by comparing the names. The comparison should always be done without paying attention to the case.

Next, a data structure, which contains the necessary information about which conversion is selected, is allocated. The data structure struct iso2022jp_data is locally defined since, outside the module, this data is not used at all. Please note that if all four conversions this modules supports are requested there are four data blocks.

One interesting thing is the initialization of the __min_ and __max_ elements of the step data object. A single ISO-2022-JP character can consist of one to four bytes. Therefore the MIN_NEEDED_FROM and MAX_NEEDED_FROM macros are defined this way. The output is always the INTERNAL character set (aka UCS-4) and therefore each character consists of exactly four bytes. For the conversion from INTERNAL to ISO-2022-JP we have to take into account that escape sequences might be necessary to switch the character sets. Therefore the __max_needed_to element for this direction gets assigned MAX_NEEDED_FROM + 2. This takes into account the two bytes needed for the escape sequences to single the switching. The asymmetry in the maximum values for the two directions can be explained easily: when reading ISO-2022-JP text, escape sequences can be handled alone (i.e., it is not necessary to process a real character since the effect of the escape sequence can be recorded in the state information). The situation is different for the other direction. Since it is in general not known which character comes next, one cannot emit escape sequences to change the state in advance. This means the escape sequences that have to be emitted together with the next character. Therefore one needs more room than only for the character itself.

The possible return values of the initialization function are:

__GCONV_OK
The initialization succeeded
__GCONV_NOCONV
The requested conversion is not supported in the module. This can happen if the gconv-modules file has errors.
__GCONV_NOMEM
Memory required to store additional information could not be allocated.

The function called before the module is unloaded is significantly easier. It often has nothing at all to do; in which case it can be left out completely.

— Data type: void (*__gconv_end_fct) (struct gconv_step *)

The task of this function is to free all resources allocated in the initialization function. Therefore only the __data element of the object pointed to by the argument is of interest. Continuing the example from the initialization function, the finalization function looks like this: 

          void
          gconv_end (struct __gconv_step *data)
          {
            free (data->__data);
          }

The most important function is the conversion function itself, which can get quite complicated for complex character sets. But since this is not of interest here, we will only describe a possible skeleton for the conversion function.

— Data type: int (*__gconv_fct) (struct __gconv_step *, struct __gconv_step_data *, const char **, const char *, size_t *, int)

The conversion function can be called for two basic reason: to convert text or to reset the state. From the description of the iconv function it can be seen why the flushing mode is necessary. What mode is selected is determined by the sixth argument, an integer. This argument being nonzero means that flushing is selected.

Common to both modes is where the output buffer can be found. The information about this buffer is stored in the conversion step data. A pointer to this information is passed as the second argument to this function. The description of the struct __gconv_step_data structure has more information on the conversion step data.

What has to be done for flushing depends on the source character set. If the source character set is not stateful, nothing has to be done. Otherwise the function has to emit a byte sequence to bring the state object into the initial state. Once this all happened the other conversion modules in the chain of conversions have to get the same chance. Whether another step follows can be determined from the __is_last element of the step data structure to which the first parameter points.

The more interesting mode is when actual text has to be converted. The first step in this case is to convert as much text as possible from the input buffer and store the result in the output buffer. The start of the input buffer is determined by the third argument, which is a pointer to a pointer variable referencing the beginning of the buffer. The fourth argument is a pointer to the byte right after the last byte in the buffer.

The conversion has to be performed according to the current state if the character set is stateful. The state is stored in an object pointed to by the __statep element of the step data (second argument). Once either the input buffer is empty or the output buffer is full the conversion stops. At this point, the pointer variable referenced by the third parameter must point to the byte following the last processed byte (i.e., if all of the input is consumed, this pointer and the fourth parameter have the same value).

What now happens depends on whether this step is the last one. If it is the last step, the only thing that has to be done is to update the __outbuf element of the step data structure to point after the last written byte. This update gives the caller the information on how much text is available in the output buffer. In addition, the variable pointed to by the fifth parameter, which is of type size_t, must be incremented by the number of characters (not bytes) that were converted in a non-reversible way. Then, the function can return.

In case the step is not the last one, the later conversion functions have to get a chance to do their work. Therefore, the appropriate conversion function has to be called. The information about the functions is stored in the conversion data structures, passed as the first parameter. This information and the step data are stored in arrays, so the next element in both cases can be found by simple pointer arithmetic: 

          int
          gconv (struct __gconv_step *step, struct __gconv_step_data *data,
                 const char **inbuf, const char *inbufend, size_t *written,
                 int do_flush)
          {
            struct __gconv_step *next_step = step + 1;
            struct __gconv_step_data *next_data = data + 1;
            ...

The next_step pointer references the next step information and next_data the next data record. The call of the next function therefore will look similar to this: 

            next_step->__fct (next_step, next_data, &outerr, outbuf,
                              written, 0)

But this is not yet all. Once the function call returns the conversion function might have some more to do. If the return value of the function is __GCONV_EMPTY_INPUT, more room is available in the output buffer. Unless the input buffer is empty the conversion, functions start all over again and process the rest of the input buffer. If the return value is not __GCONV_EMPTY_INPUT, something went wrong and we have to recover from this.

A requirement for the conversion function is that the input buffer pointer (the third argument) always point to the last character that was put in converted form into the output buffer. This is trivially true after the conversion performed in the current step, but if the conversion functions deeper downstream stop prematurely, not all characters from the output buffer are consumed and, therefore, the input buffer pointers must be backed off to the right position.

Correcting the input buffers is easy to do if the input and output character sets have a fixed width for all characters. In this situation we can compute how many characters are left in the output buffer and, therefore, can correct the input buffer pointer appropriately with a similar computation. Things are getting tricky if either character set has characters represented with variable length byte sequences, and it gets even more complicated if the conversion has to take care of the state. In these cases the conversion has to be performed once again, from the known state before the initial conversion (i.e., if necessary the state of the conversion has to be reset and the conversion loop has to be executed again). The difference now is that it is known how much input must be created, and the conversion can stop before converting the first unused character. Once this is done the input buffer pointers must be updated again and the function can return.

One final thing should be mentioned. If it is necessary for the conversion to know whether it is the first invocation (in case a prolog has to be emitted), the conversion function should increment the __invocation_counter element of the step data structure just before returning to the caller. See the description of the struct __gconv_step_data structure above for more information on how this can be used.

The return value must be one of the following values:

__GCONV_EMPTY_INPUT
All input was consumed and there is room left in the output buffer.
__GCONV_FULL_OUTPUT
No more room in the output buffer. In case this is not the last step this value is propagated down from the call of the next conversion function in the chain.
__GCONV_INCOMPLETE_INPUT
The input buffer is not entirely empty since it contains an incomplete character sequence.

The following example provides a framework for a conversion function. In case a new conversion has to be written the holes in this implementation have to be filled and that is it. 

          int
          gconv (struct __gconv_step *step, struct __gconv_step_data *data,
                 const char **inbuf, const char *inbufend, size_t *written,
                 int do_flush)
          {
            struct __gconv_step *next_step = step + 1;
            struct __gconv_step_data *next_data = data + 1;
            gconv_fct fct = next_step->__fct;
            int status;
          
            /* If the function is called with no input this means we have
               to reset to the initial state.  The possibly partly
               converted input is dropped.  */
            if (do_flush)
              {
                status = __GCONV_OK;
          
                /* Possible emit a byte sequence which put the state object
                   into the initial state.  */
          
                /* Call the steps down the chain if there are any but only
                   if we successfully emitted the escape sequence.  */
                if (status == __GCONV_OK && ! data->__is_last)
                  status = fct (next_step, next_data, NULL, NULL,
                                written, 1);
              }
            else
              {
                /* We preserve the initial values of the pointer variables.  */
                const char *inptr = *inbuf;
                char *outbuf = data->__outbuf;
                char *outend = data->__outbufend;
                char *outptr;
          
                do
                  {
                    /* Remember the start value for this round.  */
                    inptr = *inbuf;
                    /* The outbuf buffer is empty.  */
                    outptr = outbuf;
          
                    /* For stateful encodings the state must be safe here.  */
          
                    /* Run the conversion loop.  status is set
                       appropriately afterwards.  */
          
                    /* If this is the last step, leave the loop. There is
                       nothing we can do.  */
                    if (data->__is_last)
                      {
                        /* Store information about how many bytes are
                           available.  */
                        data->__outbuf = outbuf;
          
                       /* If any non-reversible conversions were performed,
                          add the number to *written.  */
          
                       break;
                     }
          
                    /* Write out all output that was produced.  */
                    if (outbuf > outptr)
                      {
                        const char *outerr = data->__outbuf;
                        int result;
          
                        result = fct (next_step, next_data, &outerr,
                                      outbuf, written, 0);
          
                        if (result != __GCONV_EMPTY_INPUT)
                          {
                            if (outerr != outbuf)
                              {
                                /* Reset the input buffer pointer.  We
                                   document here the complex case.  */
                                size_t nstatus;
          
                                /* Reload the pointers.  */
                                *inbuf = inptr;
                                outbuf = outptr;
          
                                /* Possibly reset the state.  */
          
                                /* Redo the conversion, but this time
                                   the end of the output buffer is at
                                   outerr.  */
                              }
          
                            /* Change the status.  */
                            status = result;
                          }
                        else
                          /* All the output is consumed, we can make
                              another run if everything was ok.  */
                          if (status == __GCONV_FULL_OUTPUT)
                            status = __GCONV_OK;
                     }
                  }
                while (status == __GCONV_OK);
          
                /* We finished one use of this step.  */
                ++data->__invocation_counter;
              }
          
            return status;
          }

This information should be sufficient to write new modules. Anybody doing so should also take a look at the available source code in the GNU C library sources. It contains many examples of working and optimized modules.

Next: , Previous: Character Set Handling, Up: Top

7 Locales and Internationalization

Different countries and cultures have varying conventions for how to communicate. These conventions range from very simple ones, such as the format for representing dates and times, to very complex ones, such as the language spoken.

Internationalization of software means programming it to be able to adapt to the user's favorite conventions. In ISO C, internationalization works by means of locales. Each locale specifies a collection of conventions, one convention for each purpose. The user chooses a set of conventions by specifying a locale (via environment variables).

All programs inherit the chosen locale as part of their environment. Provided the programs are written to obey the choice of locale, they will follow the conventions preferred by the user.

Next: , Up: Locales

7.1 What Effects a Locale Has

Each locale specifies conventions for several purposes, including the following:

Some aspects of adapting to the specified locale are handled automatically by the library subroutines. For example, all your program needs to do in order to use the collating sequence of the chosen locale is to use strcoll or strxfrm to compare strings.

Other aspects of locales are beyond the comprehension of the library. For example, the library can't automatically translate your program's output messages into other languages. The only way you can support output in the user's favorite language is to program this more or less by hand. The C library provides functions to handle translations for multiple languages easily.

This chapter discusses the mechanism by which you can modify the current locale. The effects of the current locale on specific library functions are discussed in more detail in the descriptions of those functions.

Next: , Previous: Effects of Locale, Up: Locales

7.2 Choosing a Locale

The simplest way for the user to choose a locale is to set the environment variable LANG. This specifies a single locale to use for all purposes. For example, a user could specify a hypothetical locale named `espana-castellano' to use the standard conventions of most of Spain.

The set of locales supported depends on the operating system you are using, and so do their names. We can't make any promises about what locales will exist, except for one standard locale called `C' or `POSIX'. Later we will describe how to construct locales.

A user also has the option of specifying different locales for different purposes—in effect, choosing a mixture of multiple locales.

For example, the user might specify the locale `espana-castellano' for most purposes, but specify the locale `usa-english' for currency formatting. This might make sense if the user is a Spanish-speaking American, working in Spanish, but representing monetary amounts in US dollars.

Note that both locales `espana-castellano' and `usa-english', like all locales, would include conventions for all of the purposes to which locales apply. However, the user can choose to use each locale for a particular subset of those purposes.

Next: , Previous: Choosing Locale, Up: Locales

7.3 Categories of Activities that Locales Affect

The purposes that locales serve are grouped into categories, so that a user or a program can choose the locale for each category independently. Here is a table of categories; each name is both an environment variable that a user can set, and a macro name that you can use as an argument to setlocale.

LC_COLLATE
This category applies to collation of strings (functions strcoll and strxfrm); see Collation Functions.
LC_CTYPE
This category applies to classification and conversion of characters, and to multibyte and wide characters; see Character Handling, and Character Set Handling.
LC_MONETARY
This category applies to formatting monetary values; see General Numeric.
LC_NUMERIC
This category applies to formatting numeric values that are not monetary; see General Numeric.
LC_TIME
This category applies to formatting date and time values; see Formatting Calendar Time.
LC_MESSAGES
This category applies to selecting the language used in the user interface for message translation (see The Uniforum approach; see Message catalogs a la X/Open) and contains regular expressions for affirmative and negative responses.
LC_ALL
This is not an environment variable; it is only a macro that you can use with setlocale to set a single locale for all purposes. Setting this environment variable overwrites all selections by the other LC_* variables or LANG.
LANG
If this environment variable is defined, its value specifies the locale to use for all purposes except as overridden by the variables above.

When developing the message translation functions it was felt that the functionality provided by the variables above is not sufficient. For example, it should be possible to specify more than one locale name. Take a Swedish user who better speaks German than English, and a program whose messages are output in English by default. It should be possible to specify that the first choice of language is Swedish, the second German, and if this also fails to use English. This is possible with the variable LANGUAGE. For further description of this GNU extension see Using gettextized software.

Next: , Previous: Locale Categories, Up: Locales

7.4 How Programs Set the Locale

A C program inherits its locale environment variables when it starts up. This happens automatically. However, these variables do not automatically control the locale used by the library functions, because ISO C says that all programs start by default in the standard `C' locale. To use the locales specified by the environment, you must call setlocale. Call it as follows: 

     setlocale (LC_ALL, "");

to select a locale based on the user choice of the appropriate environment variables.

You can also use setlocale to specify a particular locale, for general use or for a specific category.

The symbols in this section are defined in the header file locale.h.

— Function: char * setlocale (int category, const char *locale)

The function setlocale sets the current locale for category category to locale. A list of all the locales the system provides can be created by running 

            locale -a

If category is LC_ALL, this specifies the locale for all purposes. The other possible values of category specify an single purpose (see Locale Categories).

You can also use this function to find out the current locale by passing a null pointer as the locale argument. In this case, setlocale returns a string that is the name of the locale currently selected for category category.

The string returned by setlocale can be overwritten by subsequent calls, so you should make a copy of the string (see Copying and Concatenation) if you want to save it past any further calls to setlocale. (The standard library is guaranteed never to call setlocale itself.)

You should not modify the string returned by setlocale. It might be the same string that was passed as an argument in a previous call to setlocale. One requirement is that the category must be the same in the call the string was returned and the one when the string is passed in as locale parameter.

When you read the current locale for category LC_ALL, the value encodes the entire combination of selected locales for all categories. In this case, the value is not just a single locale name. In fact, we don't make any promises about what it looks like. But if you specify the same “locale name” with LC_ALL in a subsequent call to setlocale, it restores the same combination of locale selections.

To be sure you can use the returned string encoding the currently selected locale at a later time, you must make a copy of the string. It is not guaranteed that the returned pointer remains valid over time.

When the locale argument is not a null pointer, the string returned by setlocale reflects the newly-modified locale.

If you specify an empty string for locale, this means to read the appropriate environment variable and use its value to select the locale for category.

If a nonempty string is given for locale, then the locale of that name is used if possible.

If you specify an invalid locale name, setlocale returns a null pointer and leaves the current locale unchanged.

Here is an example showing how you might use setlocale to temporarily switch to a new locale. 

     #include <stddef.h>
     #include <locale.h>
     #include <stdlib.h>
     #include <string.h>
     
     void
     with_other_locale (char *new_locale,
                        void (*subroutine) (int),
                        int argument)
     {
       char *old_locale, *saved_locale;
     
       /* Get the name of the current locale.  */
       old_locale = setlocale (LC_ALL, NULL);
     
       /* Copy the name so it won't be clobbered by setlocale. */
       saved_locale = strdup (old_locale);
       if (saved_locale == NULL)
         fatal ("Out of memory");
     
       /* Now change the locale and do some stuff with it. */
       setlocale (LC_ALL, new_locale);
       (*subroutine) (argument);
     
       /* Restore the original locale. */
       setlocale (LC_ALL, saved_locale);
       free (saved_locale);
     }

Portability Note: Some ISO C systems may define additional locale categories, and future versions of the library will do so. For portability, assume that any symbol beginning with `LC_' might be defined in locale.h.

Next: , Previous: Setting the Locale, Up: Locales

7.5 Standard Locales

The only locale names you can count on finding on all operating systems are these three standard ones:

"C"
This is the standard C locale. The attributes and behavior it provides are specified in the ISO C standard. When your program starts up, it initially uses this locale by default.
"POSIX"
This is the standard POSIX locale. Currently, it is an alias for the standard C locale.
""
The empty name says to select a locale based on environment variables. See Locale Categories.

Defining and installing named locales is normally a responsibility of the system administrator at your site (or the person who installed the GNU C library). It is also possible for the user to create private locales. All this will be discussed later when describing the tool to do so.

If your program needs to use something other than the `C' locale, it will be more portable if you use whatever locale the user specifies with the environment, rather than trying to specify some non-standard locale explicitly by name. Remember, different machines might have different sets of locales installed.

Next: , Previous: Standard Locales, Up: Locales

7.6 Accessing Locale Information

There are several ways to access locale information. The simplest way is to let the C library itself do the work. Several of the functions in this library implicitly access the locale data, and use what information is provided by the currently selected locale. This is how the locale model is meant to work normally.

As an example take the strftime function, which is meant to nicely format date and time information (see Formatting Calendar Time). Part of the standard information contained in the LC_TIME category is the names of the months. Instead of requiring the programmer to take care of providing the translations the strftime function does this all by itself. %A in the format string is replaced by the appropriate weekday name of the locale currently selected by LC_TIME. This is an easy example, and wherever possible functions do things automatically in this way.

But there are quite often situations when there is simply no function to perform the task, or it is simply not possible to do the work automatically. For these cases it is necessary to access the information in the locale directly. To do this the C library provides two functions: localeconv and nl_langinfo. The former is part of ISO C and therefore portable, but has a brain-damaged interface. The second is part of the Unix interface and is portable in as far as the system follows the Unix standards.

Next: , Up: Locale Information

7.6.1 localeconv: It is portable but ...

Together with the setlocale function the ISO C people invented the localeconv function. It is a masterpiece of poor design. It is expensive to use, not extendable, and not generally usable as it provides access to only LC_MONETARY and LC_NUMERIC related information. Nevertheless, if it is applicable to a given situation it should be used since it is very portable. The function strfmon formats monetary amounts according to the selected locale using this information.

— Function: struct lconv * localeconv (void)

The localeconv function returns a pointer to a structure whose components contain information about how numeric and monetary values should be formatted in the current locale.

You should not modify the structure or its contents. The structure might be overwritten by subsequent calls to localeconv, or by calls to setlocale, but no other function in the library overwrites this value.

— Data Type: struct lconv

localeconv's return value is of this data type. Its elements are described in the following subsections.

If a member of the structure struct lconv has type char, and the value is CHAR_MAX, it means that the current locale has no value for that parameter.

Next: , Up: The Lame Way to Locale Data
7.6.1.1 Generic Numeric Formatting Parameters

These are the standard members of struct lconv; there may be others.

char *decimal_point
char *mon_decimal_point
These are the decimal-point separators used in formatting non-monetary and monetary quantities, respectively. In the `C' locale, the value of decimal_point is ".", and the value of mon_decimal_point is "".
char *thousands_sep
char *mon_thousands_sep
These are the separators used to delimit groups of digits to the left of the decimal point in formatting non-monetary and monetary quantities, respectively. In the `C' locale, both members have a value of "" (the empty string).
char *grouping
char *mon_grouping
These are strings that specify how to group the digits to the left of the decimal point. grouping applies to non-monetary quantities and mon_grouping applies to monetary quantities. Use either thousands_sep or mon_thousands_sep to separate the digit groups. Each member of these strings is to be interpreted as an integer value of type char. Successive numbers (from left to right) give the sizes of successive groups (from right to left, starting at the decimal point.) The last member is either 0, in which case the previous member is used over and over again for all the remaining groups, or CHAR_MAX, in which case there is no more grouping—or, put another way, any remaining digits form one large group without separators.

For example, if grouping is "\04\03\02", the correct grouping for the number 123456787654321 is `12', `34', `56', `78', `765', `4321'. This uses a group of 4 digits at the end, preceded by a group of 3 digits, preceded by groups of 2 digits (as many as needed). With a separator of `,', the number would be printed as `12,34,56,78,765,4321'.

A value of "\03" indicates repeated groups of three digits, as normally used in the U.S.

In the standard `C' locale, both grouping and mon_grouping have a value of "". This value specifies no grouping at all.

char int_frac_digits
char frac_digits
These are small integers indicating how many fractional digits (to the right of the decimal point) should be displayed in a monetary value in international and local formats, respectively. (Most often, both members have the same value.)

In the standard `C' locale, both of these members have the value CHAR_MAX, meaning “unspecified”. The ISO standard doesn't say what to do when you find this value; we recommend printing no fractional digits. (This locale also specifies the empty string for mon_decimal_point, so printing any fractional digits would be confusing!)

Next: , Previous: General Numeric, Up: The Lame Way to Locale Data
7.6.1.2 Printing the Currency Symbol

These members of the struct lconv structure specify how to print the symbol to identify a monetary value—the international analog of `$' for US dollars.

Each country has two standard currency symbols. The local currency symbol is used commonly within the country, while the international currency symbol is used internationally to refer to that country's currency when it is necessary to indicate the country unambiguously.

For example, many countries use the dollar as their monetary unit, and when dealing with international currencies it's important to specify that one is dealing with (say) Canadian dollars instead of U.S. dollars or Australian dollars. But when the context is known to be Canada, there is no need to make this explicit—dollar amounts are implicitly assumed to be in Canadian dollars.

char *currency_symbol
The local currency symbol for the selected locale.

In the standard `C' locale, this member has a value of "" (the empty string), meaning “unspecified”. The ISO standard doesn't say what to do when you find this value; we recommend you simply print the empty string as you would print any other string pointed to by this variable.

char *int_curr_symbol
The international currency symbol for the selected locale.

The value of int_curr_symbol should normally consist of a three-letter abbreviation determined by the international standard ISO 4217 Codes for the Representation of Currency and Funds, followed by a one-character separator (often a space).

In the standard `C' locale, this member has a value of "" (the empty string), meaning “unspecified”. We recommend you simply print the empty string as you would print any other string pointed to by this variable.

char p_cs_precedes
char n_cs_precedes
char int_p_cs_precedes
char int_n_cs_precedes
These members are 1 if the currency_symbol or int_curr_symbol strings should precede the value of a monetary amount, or 0 if the strings should follow the value. The p_cs_precedes and int_p_cs_precedes members apply to positive amounts (or zero), and the n_cs_precedes and int_n_cs_precedes members apply to negative amounts.

In the standard `C' locale, all of these members have a value of CHAR_MAX, meaning “unspecified”. The ISO standard doesn't say what to do when you find this value. We recommend printing the currency symbol before the amount, which is right for most countries. In other words, treat all nonzero values alike in these members.

The members with the int_ prefix apply to the int_curr_symbol while the other two apply to currency_symbol.

char p_sep_by_space
char n_sep_by_space
char int_p_sep_by_space
char int_n_sep_by_space
These members are 1 if a space should appear between the currency_symbol or int_curr_symbol strings and the amount, or 0 if no space should appear. The p_sep_by_space and int_p_sep_by_space members apply to positive amounts (or zero), and the n_sep_by_space and int_n_sep_by_space members apply to negative amounts.

In the standard `C' locale, all of these members have a value of CHAR_MAX, meaning “unspecified”. The ISO standard doesn't say what you should do when you find this value; we suggest you treat it as 1 (print a space). In other words, treat all nonzero values alike in these members.

The members with the int_ prefix apply to the int_curr_symbol while the other two apply to currency_symbol. There is one specialty with the int_curr_symbol, though. Since all legal values contain a space at the end the string one either printf this space (if the currency symbol must appear in front and must be separated) or one has to avoid printing this character at all (especially when at the end of the string).

Previous: Currency Symbol, Up: The Lame Way to Locale Data
7.6.1.3 Printing the Sign of a Monetary Amount

These members of the struct lconv structure specify how to print the sign (if any) of a monetary value.

char *positive_sign
char *negative_sign
These are strings used to indicate positive (or zero) and negative monetary quantities, respectively.

In the standard `C' locale, both of these members have a value of "" (the empty string), meaning “unspecified”.

The ISO standard doesn't say what to do when you find this value; we recommend printing positive_sign as you find it, even if it is empty. For a negative value, print negative_sign as you find it unless both it and positive_sign are empty, in which case print `-' instead. (Failing to indicate the sign at all seems rather unreasonable.)

char p_sign_posn
char n_sign_posn
char int_p_sign_posn
char int_n_sign_posn
These members are small integers that indicate how to position the sign for nonnegative and negative monetary quantities, respectively. (The string used by the sign is what was specified with positive_sign or negative_sign.) The possible values are as follows:
0
The currency symbol and quantity should be surrounded by parentheses.
1
Print the sign string before the quantity and currency symbol.
2
Print the sign string after the quantity and currency symbol.
3
Print the sign string right before the currency symbol.
4
Print the sign string right after the currency symbol.
CHAR_MAX
“Unspecified”. Both members have this value in the standard `C' locale.

The ISO standard doesn't say what you should do when the value is CHAR_MAX. We recommend you print the sign after the currency symbol.

The members with the int_ prefix apply to the int_curr_symbol while the other two apply to currency_symbol.

Previous: The Lame Way to Locale Data, Up: Locale Information

7.6.2 Pinpoint Access to Locale Data

When writing the X/Open Portability Guide the authors realized that the localeconv function is not enough to provide reasonable access to locale information. The information which was meant to be available in the locale (as later specified in the POSIX.1 standard) requires more ways to access it. Therefore the nl_langinfo function was introduced.

— Function: char * nl_langinfo (nl_item item)

The nl_langinfo function can be used to access individual elements of the locale categories. Unlike the localeconv function, which returns all the information, nl_langinfo lets the caller select what information it requires. This is very fast and it is not a problem to call this function multiple times.

A second advantage is that in addition to the numeric and monetary formatting information, information from the LC_TIME and LC_MESSAGES categories is available.

The type nl_type is defined in nl_types.h. The argument item is a numeric value defined in the header langinfo.h. The X/Open standard defines the following values:

CODESET
nl_langinfo returns a string with the name of the coded character set used in the selected locale.
ABDAY_1
ABDAY_2
ABDAY_3
ABDAY_4
ABDAY_5
ABDAY_6
ABDAY_7
nl_langinfo returns the abbreviated weekday name. ABDAY_1 corresponds to Sunday.
DAY_1
DAY_2
DAY_3
DAY_4
DAY_5
DAY_6
DAY_7
Similar to ABDAY_1 etc., but here the return value is the unabbreviated weekday name.
ABMON_1
ABMON_2
ABMON_3
ABMON_4
ABMON_5
ABMON_6
ABMON_7
ABMON_8
ABMON_9
ABMON_10
ABMON_11
ABMON_12
The return value is abbreviated name of the month. ABMON_1 corresponds to January.
MON_1
MON_2
MON_3
MON_4
MON_5
MON_6
MON_7
MON_8
MON_9
MON_10
MON_11
MON_12
Similar to ABMON_1 etc., but here the month names are not abbreviated. Here the first value MON_1 also corresponds to January.
AM_STR
PM_STR
The return values are strings which can be used in the representation of time as an hour from 1 to 12 plus an am/pm specifier.

Note that in locales which do not use this time representation these strings might be empty, in which case the am/pm format cannot be used at all.

D_T_FMT
The return value can be used as a format string for strftime to represent time and date in a locale-specific way.
D_FMT
The return value can be used as a format string for strftime to represent a date in a locale-specific way.
T_FMT
The return value can be used as a format string for strftime to represent time in a locale-specific way.
T_FMT_AMPM
The return value can be used as a format string for strftime to represent time in the am/pm format.

Note that if the am/pm format does not make any sense for the selected locale, the return value might be the same as the one for T_FMT.

ERA
The return value represents the era used in the current locale.

Most locales do not define this value. An example of a locale which does define this value is the Japanese one. In Japan, the traditional representation of dates includes the name of the era corresponding to the then-emperor's reign.

Normally it should not be necessary to use this value directly. Specifying the E modifier in their format strings causes the strftime functions to use this information. The format of the returned string is not specified, and therefore you should not assume knowledge of it on different systems.

ERA_YEAR
The return value gives the year in the relevant era of the locale. As for ERA it should not be necessary to use this value directly.
ERA_D_T_FMT
This return value can be used as a format string for strftime to represent dates and times in a locale-specific era-based way.
ERA_D_FMT
This return value can be used as a format string for strftime to represent a date in a locale-specific era-based way.
ERA_T_FMT
This return value can be used as a format string for strftime to represent time in a locale-specific era-based way.
ALT_DIGITS
The return value is a representation of up to 100 values used to represent the values 0 to 99. As for ERA this value is not intended to be used directly, but instead indirectly through the strftime function. When the modifier O is used in a format which would otherwise use numerals to represent hours, minutes, seconds, weekdays, months, or weeks, the appropriate value for the locale is used instead.
INT_CURR_SYMBOL
The same as the value returned by localeconv in the int_curr_symbol element of the struct lconv.
CURRENCY_SYMBOL
CRNCYSTR
The same as the value returned by localeconv in the currency_symbol element of the struct lconv.

CRNCYSTR is a deprecated alias still required by Unix98.

MON_DECIMAL_POINT
The same as the value returned by localeconv in the mon_decimal_point element of the struct lconv.
MON_THOUSANDS_SEP
The same as the value returned by localeconv in the mon_thousands_sep element of the struct lconv.
MON_GROUPING
The same as the value returned by localeconv in the mon_grouping element of the struct lconv.
POSITIVE_SIGN
The same as the value returned by localeconv in the positive_sign element of the struct lconv.
NEGATIVE_SIGN
The same as the value returned by localeconv in the negative_sign element of the struct lconv.
INT_FRAC_DIGITS
The same as the value returned by localeconv in the int_frac_digits element of the struct lconv.
FRAC_DIGITS
The same as the value returned by localeconv in the frac_digits element of the struct lconv.
P_CS_PRECEDES
The same as the value returned by localeconv in the p_cs_precedes element of the struct lconv.
P_SEP_BY_SPACE
The same as the value returned by localeconv in the p_sep_by_space element of the struct lconv.
N_CS_PRECEDES
The same as the value returned by localeconv in the n_cs_precedes element of the struct lconv.
N_SEP_BY_SPACE
The same as the value returned by localeconv in the n_sep_by_space element of the struct lconv.
P_SIGN_POSN
The same as the value returned by localeconv in the p_sign_posn element of the struct lconv.
N_SIGN_POSN
The same as the value returned by localeconv in the n_sign_posn element of the struct lconv.
INT_P_CS_PRECEDES
The same as the value returned by localeconv in the int_p_cs_precedes element of the struct lconv.
INT_P_SEP_BY_SPACE
The same as the value returned by localeconv in the int_p_sep_by_space element of the struct lconv.
INT_N_CS_PRECEDES
The same as the value returned by localeconv in the int_n_cs_precedes element of the struct lconv.
INT_N_SEP_BY_SPACE
The same as the value returned by localeconv in the int_n_sep_by_space element of the struct lconv.
INT_P_SIGN_POSN
The same as the value returned by localeconv in the int_p_sign_posn element of the struct lconv.
INT_N_SIGN_POSN
The same as the value returned by localeconv in the int_n_sign_posn element of the struct lconv.
DECIMAL_POINT
RADIXCHAR
The same as the value returned by localeconv in the decimal_point element of the struct lconv.

The name RADIXCHAR is a deprecated alias still used in Unix98.

THOUSANDS_SEP
THOUSEP
The same as the value returned by localeconv in the thousands_sep element of the struct lconv.

The name THOUSEP is a deprecated alias still used in Unix98.

GROUPING
The same as the value returned by localeconv in the grouping element of the struct lconv.
YESEXPR
The return value is a regular expression which can be used with the regex function to recognize a positive response to a yes/no question. The GNU C library provides the rpmatch function for easier handling in applications.
NOEXPR
The return value is a regular expression which can be used with the regex function to recognize a negative response to a yes/no question.
YESSTR
The return value is a locale-specific translation of the positive response to a yes/no question.

Using this value is deprecated since it is a very special case of message translation, and is better handled by the message translation functions (see Message Translation).

The use of this symbol is deprecated. Instead message translation should be used.

NOSTR
The return value is a locale-specific translation of the negative response to a yes/no question. What is said for YESSTR is also true here.

The use of this symbol is deprecated. Instead message translation should be used.

The file langinfo.h defines a lot more symbols but none of them is official. Using them is not portable, and the format of the return values might change. Therefore we recommended you not use them.

Note that the return value for any valid argument can be used for in all situations (with the possible exception of the am/pm time formatting codes). If the user has not selected any locale for the appropriate category, nl_langinfo returns the information from the "C" locale. It is therefore possible to use this function as shown in the example below.

If the argument item is not valid, a pointer to an empty string is returned.

An example of nl_langinfo usage is a function which has to print a given date and time in a locale-specific way. At first one might think that, since strftime internally uses the locale information, writing something like the following is enough: 

     size_t
     i18n_time_n_data (char *s, size_t len, const struct tm *tp)
     {
       return strftime (s, len, "%X %D", tp);
     }

The format contains no weekday or month names and therefore is internationally usable. Wrong! The output produced is something like "hh:mm:ss MM/DD/YY". This format is only recognizable in the USA. Other countries use different formats. Therefore the function should be rewritten like this: 

     size_t
     i18n_time_n_data (char *s, size_t len, const struct tm *tp)
     {
       return strftime (s, len, nl_langinfo (D_T_FMT), tp);
     }

Now it uses the date and time format of the locale selected when the program runs. If the user selects the locale correctly there should never be a misunderstanding over the time and date format.

Next: , Previous: Locale Information, Up: Locales

7.7 A dedicated function to format numbers

We have seen that the structure returned by localeconv as well as the values given to nl_langinfo allow you to retrieve the various pieces of locale-specific information to format numbers and monetary amounts. We have also seen that the underlying rules are quite complex.

Therefore the X/Open standards introduce a function which uses such locale information, making it easier for the user to format numbers according to these rules.

— Function: ssize_t strfmon (char *s, size_t maxsize, const char *format, ...)

The strfmon function is similar to the strftime function in that it takes a buffer, its size, a format string, and values to write into the buffer as text in a form specified by the format string. Like strftime, the function also returns the number of bytes written into the buffer.

There are two differences: strfmon can take more than one argument, and, of course, the format specification is different. Like strftime, the format string consists of normal text, which is output as is, and format specifiers, which are indicated by a `%'. Immediately after the `%', you can optionally specify various flags and formatting information before the main formatting character, in a similar way to printf:

The next part of a specification is an optional field width. If no width is specified 0 is taken. During output, the function first determines how much space is required. If it requires at least as many characters as given by the field width, it is output using as much space as necessary. Otherwise, it is extended to use the full width by filling with the space character. The presence or absence of the `-' flag determines the side at which such padding occurs. If present, the spaces are added at the right making the output left-justified, and vice versa.

So far the format looks familiar, being similar to the printf and strftime formats. However, the next two optional fields introduce something new. The first one is a `#' character followed by a decimal digit string. The value of the digit string specifies the number of digit positions to the left of the decimal point (or equivalent). This does not include the grouping character when the `^' flag is not given. If the space needed to print the number does not fill the whole width, the field is padded at the left side with the fill character, which can be selected using the `=' flag and by default is a space. For example, if the field width is selected as 6 and the number is 123, the fill character is `*' the result will be `***123'.

The second optional field starts with a `.' (period) and consists of another decimal digit string. Its value describes the number of characters printed after the decimal point. The default is selected from the current locale (frac_digits, int_frac_digits, see see General Numeric). If the exact representation needs more digits than given by the field width, the displayed value is rounded. If the number of fractional digits is selected to be zero, no decimal point is printed.

As a GNU extension, the strfmon implementation in the GNU libc allows an optional `L' next as a format modifier. If this modifier is given, the argument is expected to be a long double instead of a double value.

Finally, the last component is a format specifier. There are three specifiers defined:

`i'
Use the locale's rules for formatting an international currency value.
`n'
Use the locale's rules for formatting a national currency value.
`%'
Place a `%' in the output. There must be no flag, width specifier or modifier given, only `%%' is allowed.

As for printf, the function reads the format string from left to right and uses the values passed to the function following the format string. The values are expected to be either of type double or long double, depending on the presence of the modifier `L'. The result is stored in the buffer pointed to by s. At most maxsize characters are stored.

The return value of the function is the number of characters stored in s, including the terminating NULL byte. If the number of characters stored would exceed maxsize, the function returns -1 and the content of the buffer s is unspecified. In this case errno is set to E2BIG.

A few examples should make clear how the function works. It is assumed that all the following pieces of code are executed in a program which uses the USA locale (en_US). The simplest form of the format is this: 

     strfmon (buf, 100, "@%n@%n@%n@", 123.45, -567.89, 12345.678);

The output produced is 

     "@$123.45@-$567.89@$12,345.68@"

We can notice several things here. First, the widths of the output numbers are different. We have not specified a width in the format string, and so this is no wonder. Second, the third number is printed using thousands separators. The thousands separator for the en_US locale is a comma. The number is also rounded. .678 is rounded to .68 since the format does not specify a precision and the default value in the locale is 2. Finally, note that the national currency symbol is printed since `%n' was used, not `i'. The next example shows how we can align the output. 

     strfmon (buf, 100, "@%=*11n@%=*11n@%=*11n@", 123.45, -567.89, 12345.678);

The output this time is: 

     "@    $123.45@   -$567.89@ $12,345.68@"

Two things stand out. Firstly, all fields have the same width (eleven characters) since this is the width given in the format and since no number required more characters to be printed. The second important point is that the fill character is not used. This is correct since the white space was not used to achieve a precision given by a `#' modifier, but instead to fill to the given width. The difference becomes obvious if we now add a width specification. 

     strfmon (buf, 100, "@%=*11#5n@%=*11#5n@%=*11#5n@",
              123.45, -567.89, 12345.678);

The output is 

     "@ $***123.45@-$***567.89@ $12,456.68@"

Here we can see that all the currency symbols are now aligned, and that the space between the currency sign and the number is filled with the selected fill character. Note that although the width is selected to be 5 and 123.45 has three digits left of the decimal point, the space is filled with three asterisks. This is correct since, as explained above, the width does not include the positions used to store thousands separators. One last example should explain the remaining functionality. 

     strfmon (buf, 100, "@%=0(16#5.3i@%=0(16#5.3i@%=0(16#5.3i@",
              123.45, -567.89, 12345.678);

This rather complex format string produces the following output: 

     "@ USD 000123,450 @(USD 000567.890)@ USD 12,345.678 @"

The most noticeable change is the alternative way of representing negative numbers. In financial circles this is often done using parentheses, and this is what the `(' flag selected. The fill character is now `0'. Note that this `0' character is not regarded as a numeric zero, and therefore the first and second numbers are not printed using a thousands separator. Since we used the format specifier `i' instead of `n', the international form of the currency symbol is used. This is a four letter string, in this case "USD ". The last point is that since the precision right of the decimal point is selected to be three, the first and second numbers are printed with an extra zero at the end and the third number is printed without rounding.

Previous: Formatting Numbers, Up: Locales

7.8 Yes-or-No Questions

Some non GUI programs ask a yes-or-no question. If the messages (especially the questions) are translated into foreign languages, be sure that you localize the answers too. It would be very bad habit to ask a question in one language and request the answer in another, often English.

The GNU C library contains rpmatch to give applications easy access to the corresponding locale definitions.

— Function: int rpmatch (const char *response)

The function rpmatch checks the string in response whether or not it is a correct yes-or-no answer and if yes, which one. The check uses the YESEXPR and NOEXPR data in the LC_MESSAGES category of the currently selected locale. The return value is as follows:

1
The user entered an affirmative answer.
0
The user entered a negative answer.
-1
The answer matched neither the YESEXPR nor the NOEXPR regular expression.

This function is not standardized but available beside in GNU libc at least also in the IBM AIX library.

This function would normally be used like this: 

       ...
       /* Use a safe default.  */
       _Bool doit = false;
     
       fputs (gettext ("Do you really want to do this? "), stdout);
       fflush (stdout);
       /* Prepare the getline call.  */
       line = NULL;
       len = 0;
       while (getline (&line, &len, stdout) >= 0)
         {
           /* Check the response.  */
           int res = rpmatch (line);
           if (res >= 0)
             {
               /* We got a definitive answer.  */
               if (res > 0)
                 doit = true;
               break;
             }
         }
       /* Free what getline allocated.  */
       free (line);

Note that the loop continues until an read error is detected or until a definitive (positive or negative) answer is read.

Next: , Previous: Locales, Up: Top

8 Message Translation

The program's interface with the human should be designed in a way to ease the human the task. One of the possibilities is to use messages in whatever language the user prefers.

Printing messages in different languages can be implemented in different ways. One could add all the different languages in the source code and add among the variants every time a message has to be printed. This is certainly no good solution since extending the set of languages is difficult (the code must be changed) and the code itself can become really big with dozens of message sets.

A better solution is to keep the message sets for each language are kept in separate files which are loaded at runtime depending on the language selection of the user.

The GNU C Library provides two different sets of functions to support message translation. The problem is that neither of the interfaces is officially defined by the POSIX standard. The catgets family of functions is defined in the X/Open standard but this is derived from industry decisions and therefore not necessarily based on reasonable decisions.

As mentioned above the message catalog handling provides easy extendibility by using external data files which contain the message translations. I.e., these files contain for each of the messages used in the program a translation for the appropriate language. So the tasks of the message handling functions are

The two approaches mainly differ in the implementation of this last step. The design decisions made for this influences the whole rest.

Next: , Up: Message Translation

8.1 X/Open Message Catalog Handling

The catgets functions are based on the simple scheme:

Associate every message to translate in the source code with a unique identifier. To retrieve a message from a catalog file solely the identifier is used.

This means for the author of the program that s/he will have to make sure the meaning of the identifier in the program code and in the message catalogs are always the same.

Before a message can be translated the catalog file must be located. The user of the program must be able to guide the responsible function to find whatever catalog the user wants. This is separated from what the programmer had in mind.

All the types, constants and functions for the catgets functions are defined/declared in the nl_types.h header file.

Next: , Up: Message catalogs a la X/Open

8.1.1 The catgets function family

— Function: nl_catd catopen (const char *cat_name, int flag)

The catgets function tries to locate the message data file names cat_name and loads it when found. The return value is of an opaque type and can be used in calls to the other functions to refer to this loaded catalog.

The return value is (nl_catd) -1 in case the function failed and no catalog was loaded. The global variable errno contains a code for the error causing the failure. But even if the function call succeeded this does not mean that all messages can be translated.

Locating the catalog file must happen in a way which lets the user of the program influence the decision. It is up to the user to decide about the language to use and sometimes it is useful to use alternate catalog files. All this can be specified by the user by setting some environment variables.

The first problem is to find out where all the message catalogs are stored. Every program could have its own place to keep all the different files but usually the catalog files are grouped by languages and the catalogs for all programs are kept in the same place.

To tell the catopen function where the catalog for the program can be found the user can set the environment variable NLSPATH to a value which describes her/his choice. Since this value must be usable for different languages and locales it cannot be a simple string. Instead it is a format string (similar to printf's). An example is 

          /usr/share/locale/%L/%N:/usr/share/locale/%L/LC_MESSAGES/%N

First one can see that more than one directory can be specified (with the usual syntax of separating them by colons). The next things to observe are the format string, %L and %N in this case. The catopen function knows about several of them and the replacement for all of them is of course different.

%N
This format element is substituted with the name of the catalog file. This is the value of the cat_name argument given to catgets.
%L
This format element is substituted with the name of the currently selected locale for translating messages. How this is determined is explained below.
%l
(This is the lowercase ell.) This format element is substituted with the language element of the locale name. The string describing the selected locale is expected to have the form lang[_terr[.codeset]] and this format uses the first part lang.
%t
This format element is substituted by the territory part terr of the name of the currently selected locale. See the explanation of the format above.
%c
This format element is substituted by the codeset part codeset of the name of the currently selected locale. See the explanation of the format above.
%%
Since % is used in a meta character there must be a way to express the % character in the result itself. Using %% does this just like it works for printf.

Using NLSPATH allows arbitrary directories to be searched for message catalogs while still allowing different languages to be used. If the NLSPATH environment variable is not set, the default value is 

          prefix/share/locale/%L/%N:prefix/share/locale/%L/LC_MESSAGES/%N

where prefix is given to configure while installing the GNU C Library (this value is in many cases /usr or the empty string).

The remaining problem is to decide which must be used. The value decides about the substitution of the format elements mentioned above. First of all the user can specify a path in the message catalog name (i.e., the name contains a slash character). In this situation the NLSPATH environment variable is not used. The catalog must exist as specified in the program, perhaps relative to the current working directory. This situation in not desirable and catalogs names never should be written this way. Beside this, this behavior is not portable to all other platforms providing the catgets interface.

Otherwise the values of environment variables from the standard environment are examined (see Standard Environment). Which variables are examined is decided by the flag parameter of catopen. If the value is NL_CAT_LOCALE (which is defined in nl_types.h) then the catopen function use the name of the locale currently selected for the LC_MESSAGES category.

If flag is zero the LANG environment variable is examined. This is a left-over from the early days where the concept of the locales had not even reached the level of POSIX locales.

The environment variable and the locale name should have a value of the form lang[_terr[.codeset]] as explained above. If no environment variable is set the "C" locale is used which prevents any translation.

The return value of the function is in any case a valid string. Either it is a translation from a message catalog or it is the same as the string parameter. So a piece of code to decide whether a translation actually happened must look like this: 

          {
            char *trans = catgets (desc, set, msg, input_string);
            if (trans == input_string)
              {
                /* Something went wrong.  */
              }
          }

When an error occurred the global variable errno is set to

EBADF
The catalog does not exist.
ENOMSG
The set/message tuple does not name an existing element in the message catalog.

While it sometimes can be useful to test for errors programs normally will avoid any test. If the translation is not available it is no big problem if the original, untranslated message is printed. Either the user understands this as well or s/he will look for the reason why the messages are not translated.

Please note that the currently selected locale does not depend on a call to the setlocale function. It is not necessary that the locale data files for this locale exist and calling setlocale succeeds. The catopen function directly reads the values of the environment variables.

— Function: char * catgets (nl_catd catalog_desc, int set, int message, const char *string)

The function catgets has to be used to access the massage catalog previously opened using the catopen function. The catalog_desc parameter must be a value previously returned by catopen.

The next two parameters, set and message, reflect the internal organization of the message catalog files. This will be explained in detail below. For now it is interesting to know that a catalog can consists of several set and the messages in each thread are individually numbered using numbers. Neither the set number nor the message number must be consecutive. They can be arbitrarily chosen. But each message (unless equal to another one) must have its own unique pair of set and message number.

Since it is not guaranteed that the message catalog for the language selected by the user exists the last parameter string helps to handle this case gracefully. If no matching string can be found string is returned. This means for the programmer that

It is somewhat uncomfortable to write a program using the catgets functions if no supporting functionality is available. Since each set/message number tuple must be unique the programmer must keep lists of the messages at the same time the code is written. And the work between several people working on the same project must be coordinated. We will see some how these problems can be relaxed a bit (see Common Usage).

— Function: int catclose (nl_catd catalog_desc)

The catclose function can be used to free the resources associated with a message catalog which previously was opened by a call to catopen. If the resources can be successfully freed the function returns 0. Otherwise it return −1 and the global variable errno is set. Errors can occur if the catalog descriptor catalog_desc is not valid in which case errno is set to EBADF.

Next: , Previous: The catgets Functions, Up: Message catalogs a la X/Open

8.1.2 Format of the message catalog files

The only reasonable way the translate all the messages of a function and store the result in a message catalog file which can be read by the catopen function is to write all the message text to the translator and let her/him translate them all. I.e., we must have a file with entries which associate the set/message tuple with a specific translation. This file format is specified in the X/Open standard and is as follows:

Important: The handling of identifiers instead of numbers for the set and messages is a GNU extension. Systems strictly following the X/Open specification do not have this feature. An example for a message catalog file is this: 

     $ This is a leading comment.
     $quote "
     
     $set SetOne
     1 Message with ID 1.
     two "   Message with ID \"two\", which gets the value 2 assigned"
     
     $set SetTwo
     $ Since the last set got the number 1 assigned this set has number 2.
     4000 "The numbers can be arbitrary, they need not start at one."

This small example shows various aspects:

While this file format is pretty easy it is not the best possible for use in a running program. The catopen function would have to parser the file and handle syntactic errors gracefully. This is not so easy and the whole process is pretty slow. Therefore the catgets functions expect the data in another more compact and ready-to-use file format. There is a special program gencat which is explained in detail in the next section.

Files in this other format are not human readable. To be easy to use by programs it is a binary file. But the format is byte order independent so translation files can be shared by systems of arbitrary architecture (as long as they use the GNU C Library).

Details about the binary file format are not important to know since these files are always created by the gencat program. The sources of the GNU C Library also provide the sources for the gencat program and so the interested reader can look through these source files to learn about the file format.

Next: , Previous: The message catalog files, Up: Message catalogs a la X/Open

8.1.3 Generate Message Catalogs files

The gencat program is specified in the X/Open standard and the GNU implementation follows this specification and so processes all correctly formed input files. Additionally some extension are implemented which help to work in a more reasonable way with the catgets functions.

The gencat program can be invoked in two ways: 

     `gencat [Option]... [Output-File [Input-File]...]`

This is the interface defined in the X/Open standard. If no Input-File parameter is given input will be read from standard input. Multiple input files will be read as if they are concatenated. If Output-File is also missing, the output will be written to standard output. To provide the interface one is used to from other programs a second interface is provided. 

     `gencat [Option]... -o Output-File [Input-File]...`

The option `-o' is used to specify the output file and all file arguments are used as input files.

Beside this one can use - or /dev/stdin for Input-File to denote the standard input. Corresponding one can use - and /dev/stdout for Output-File to denote standard output. Using - as a file name is allowed in X/Open while using the device names is a GNU extension.

The gencat program works by concatenating all input files and then merge the resulting collection of message sets with a possibly existing output file. This is done by removing all messages with set/message number tuples matching any of the generated messages from the output file and then adding all the new messages. To regenerate a catalog file while ignoring the old contents therefore requires to remove the output file if it exists. If the output is written to standard output no merging takes place.

The following table shows the options understood by the gencat program. The X/Open standard does not specify any option for the program so all of these are GNU extensions.

`-V'
`--version'
Print the version information and exit.
`-h'
`--help'
Print a usage message listing all available options, then exit successfully.
`--new'
Do never merge the new messages from the input files with the old content of the output files. The old content of the output file is discarded.
`-H'
`--header=name'
This option is used to emit the symbolic names given to sets and messages in the input files for use in the program. Details about how to use this are given in the next section. The name parameter to this option specifies the name of the output file. It will contain a number of C preprocessor #defines to associate a name with a number.

Please note that the generated file only contains the symbols from the input files. If the output is merged with the previous content of the output file the possibly existing symbols from the file(s) which generated the old output files are not in the generated header file.

Previous: The gencat program, Up: Message catalogs a la X/Open

8.1.4 How to use the catgets interface

The catgets functions can be used in two different ways. By following slavishly the X/Open specs and not relying on the extension and by using the GNU extensions. We will take a look at the former method first to understand the benefits of extensions.

8.1.4.1 Not using symbolic names

Since the X/Open format of the message catalog files does not allow symbol names we have to work with numbers all the time. When we start writing a program we have to replace all appearances of translatable strings with something like 

     catgets (catdesc, set, msg, "string")

catgets is retrieved from a call to catopen which is normally done once at the program start. The "string" is the string we want to translate. The problems start with the set and message numbers.

In a bigger program several programmers usually work at the same time on the program and so coordinating the number allocation is crucial. Though no two different strings must be indexed by the same tuple of numbers it is highly desirable to reuse the numbers for equal strings with equal translations (please note that there might be strings which are equal in one language but have different translations due to difference contexts).

The allocation process can be relaxed a bit by different set numbers for different parts of the program. So the number of developers who have to coordinate the allocation can be reduced. But still lists must be keep track of the allocation and errors can easily happen. These errors cannot be discovered by the compiler or the catgets functions. Only the user of the program might see wrong messages printed. In the worst cases the messages are so irritating that they cannot be recognized as wrong. Think about the translations for "true" and "false" being exchanged. This could result in a disaster.

8.1.4.2 Using symbolic names

The problems mentioned in the last section derive from the fact that:

  1. the numbers are allocated once and due to the possibly frequent use of them it is difficult to change a number later.
  2. the numbers do not allow to guess anything about the string and therefore collisions can easily happen.

By constantly using symbolic names and by providing a method which maps the string content to a symbolic name (however this will happen) one can prevent both problems above. The cost of this is that the programmer has to write a complete message catalog file while s/he is writing the program itself.

This is necessary since the symbolic names must be mapped to numbers before the program sources can be compiled. In the last section it was described how to generate a header containing the mapping of the names. E.g., for the example message file given in the last section we could call the gencat program as follow (assume ex.msg contains the sources). 

     gencat -H ex.h -o ex.cat ex.msg

This generates a header file with the following content: 

     #define SetTwoSet 0x2   /* ex.msg:8 */
     
     #define SetOneSet 0x1   /* ex.msg:4 */
     #define SetOnetwo 0x2   /* ex.msg:6 */

As can be seen the various symbols given in the source file are mangled to generate unique identifiers and these identifiers get numbers assigned. Reading the source file and knowing about the rules will allow to predict the content of the header file (it is deterministic) but this is not necessary. The gencat program can take care for everything. All the programmer has to do is to put the generated header file in the dependency list of the source files of her/his project and to add a rules to regenerate the header of any of the input files change.

One word about the symbol mangling. Every symbol consists of two parts: the name of the message set plus the name of the message or the special string Set. So SetOnetwo means this macro can be used to access the translation with identifier two in the message set SetOne.

The other names denote the names of the message sets. The special string Set is used in the place of the message identifier.

If in the code the second string of the set SetOne is used the C code should look like this: 

     catgets (catdesc, SetOneSet, SetOnetwo,
              "   Message with ID \"two\", which gets the value 2 assigned")

Writing the function this way will allow to change the message number and even the set number without requiring any change in the C source code. (The text of the string is normally not the same; this is only for this example.)

8.1.4.3 How does to this allow to develop

To illustrate the usual way to work with the symbolic version numbers here is a little example. Assume we want to write the very complex and famous greeting program. We start by writing the code as usual: 

     #include <stdio.h>
     int
     main (void)
     {
       printf ("Hello, world!\n");
       return 0;
     }

Now we want to internationalize the message and therefore replace the message with whatever the user wants. 

     #include <nl_types.h>
     #include <stdio.h>
     #include "msgnrs.h"
     int
     main (void)
     {
       nl_catd catdesc = catopen ("hello.cat", NL_CAT_LOCALE);
       printf (catgets (catdesc, SetMainSet, SetMainHello,
                        "Hello, world!\n"));
       catclose (catdesc);
       return 0;
     }

We see how the catalog object is opened and the returned descriptor used in the other function calls. It is not really necessary to check for failure of any of the functions since even in these situations the functions will behave reasonable. They simply will be return a translation.

What remains unspecified here are the constants SetMainSet and SetMainHello. These are the symbolic names describing the message. To get the actual definitions which match the information in the catalog file we have to create the message catalog source file and process it using the gencat program. 

     $ Messages for the famous greeting program.
     $quote "
     
     $set Main
     Hello "Hallo, Welt!\n"

Now we can start building the program (assume the message catalog source file is named hello.msg and the program source file hello.c): 

     

     % gencat -H msgnrs.h -o hello.cat hello.msg
     % cat msgnrs.h
     #define MainSet 0x1     /* hello.msg:4 */
     #define MainHello 0x1   /* hello.msg:5 */
     % gcc -o hello hello.c -I.
     % cp hello.cat /usr/share/locale/de/LC_MESSAGES
     % echo $LC_ALL
     de
     % ./hello
     Hallo, Welt!
     %

The call of the gencat program creates the missing header file msgnrs.h as well as the message catalog binary. The former is used in the compilation of hello.c while the later is placed in a directory in which the catopen function will try to locate it. Please check the LC_ALL environment variable and the default path for catopen presented in the description above.

Previous: Message catalogs a la X/Open, Up: Message Translation

8.2 The Uniforum approach to Message Translation

Sun Microsystems tried to standardize a different approach to message translation in the Uniforum group. There never was a real standard defined but still the interface was used in Sun's operation systems. Since this approach fits better in the development process of free software it is also used throughout the GNU project and the GNU gettext package provides support for this outside the GNU C Library.

The code of the libintl from GNU gettext is the same as the code in the GNU C Library. So the documentation in the GNU gettext manual is also valid for the functionality here. The following text will describe the library functions in detail. But the numerous helper programs are not described in this manual. Instead people should read the GNU gettext manual (see GNU gettext utilities). We will only give a short overview.

Though the catgets functions are available by default on more systems the gettext interface is at least as portable as the former. The GNU gettext package can be used wherever the functions are not available.

Next: , Up: The Uniforum approach

8.2.1 The gettext family of functions

The paradigms underlying the gettext approach to message translations is different from that of the catgets functions the basic functionally is equivalent. There are functions of the following categories:

Next: , Up: Message catalogs with gettext
8.2.1.1 What has to be done to translate a message?

The gettext functions have a very simple interface. The most basic function just takes the string which shall be translated as the argument and it returns the translation. This is fundamentally different from the catgets approach where an extra key is necessary and the original string is only used for the error case.

If the string which has to be translated is the only argument this of course means the string itself is the key. I.e., the translation will be selected based on the original string. The message catalogs must therefore contain the original strings plus one translation for any such string. The task of the gettext function is it to compare the argument string with the available strings in the catalog and return the appropriate translation. Of course this process is optimized so that this process is not more expensive than an access using an atomic key like in catgets.

The gettext approach has some advantages but also some disadvantages. Please see the GNU gettext manual for a detailed discussion of the pros and cons.

All the definitions and declarations for gettext can be found in the libintl.h header file. On systems where these functions are not part of the C library they can be found in a separate library named libintl.a (or accordingly different for shared libraries).

— Function: char * gettext (const char *msgid)

The gettext function searches the currently selected message catalogs for a string which is equal to msgid. If there is such a string available it is returned. Otherwise the argument string msgid is returned.

Please note that all though the return value is char * the returned string must not be changed. This broken type results from the history of the function and does not reflect the way the function should be used.

Please note that above we wrote “message catalogs” (plural). This is a specialty of the GNU implementation of these functions and we will say more about this when we talk about the ways message catalogs are selected (see Locating gettext catalog).

The gettext function does not modify the value of the global errno variable. This is necessary to make it possible to write something like 

            printf (gettext ("Operation failed: %m\n"));

Here the errno value is used in the printf function while processing the %m format element and if the gettext function would change this value (it is called before printf is called) we would get a wrong message.

So there is no easy way to detect a missing message catalog beside comparing the argument string with the result. But it is normally the task of the user to react on missing catalogs. The program cannot guess when a message catalog is really necessary since for a user who speaks the language the program was developed in does not need any translation.

The remaining two functions to access the message catalog add some functionality to select a message catalog which is not the default one. This is important if parts of the program are developed independently. Every part can have its own message catalog and all of them can be used at the same time. The C library itself is an example: internally it uses the gettext functions but since it must not depend on a currently selected default message catalog it must specify all ambiguous information.

— Function: char * dgettext (const char *domainname, const char *msgid)

The dgettext functions acts just like the gettext function. It only takes an additional first argument domainname which guides the selection of the message catalogs which are searched for the translation. If the domainname parameter is the null pointer the dgettext function is exactly equivalent to gettext since the default value for the domain name is used.

As for gettext the return value type is char * which is an anachronism. The returned string must never be modified.

— Function: char * dcgettext (const char *domainname, const char *msgid, int category)

The dcgettext adds another argument to those which dgettext takes. This argument category specifies the last piece of information needed to localize the message catalog. I.e., the domain name and the locale category exactly specify which message catalog has to be used (relative to a given directory, see below).

The dgettext function can be expressed in terms of dcgettext by using 

          dcgettext (domain, string, LC_MESSAGES)

instead of 

          dgettext (domain, string)

This also shows which values are expected for the third parameter. One has to use the available selectors for the categories available in locale.h. Normally the available values are LC_CTYPE, LC_COLLATE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC, and LC_TIME. Please note that LC_ALL must not be used and even though the names might suggest this, there is no relation to the environments variables of this name.

The dcgettext function is only implemented for compatibility with other systems which have gettext functions. There is not really any situation where it is necessary (or useful) to use a different value but LC_MESSAGES in for the category parameter. We are dealing with messages here and any other choice can only be irritating.

As for gettext the return value type is char * which is an anachronism. The returned string must never be modified.

When using the three functions above in a program it is a frequent case that the msgid argument is a constant string. So it is worth to optimize this case. Thinking shortly about this one will realize that as long as no new message catalog is loaded the translation of a message will not change. This optimization is actually implemented by the gettext, dgettext and dcgettext functions.

Next: , Previous: Translation with gettext, Up: Message catalogs with gettext
8.2.1.2 How to determine which catalog to be used

The functions to retrieve the translations for a given message have a remarkable simple interface. But to provide the user of the program still the opportunity to select exactly the translation s/he wants and also to provide the programmer the possibility to influence the way to locate the search for catalogs files there is a quite complicated underlying mechanism which controls all this. The code is complicated the use is easy.

Basically we have two different tasks to perform which can also be performed by the catgets functions:

  1. Locate the set of message catalogs. There are a number of files for different languages and which all belong to the package. Usually they are all stored in the filesystem below a certain directory.

    There can be arbitrary many packages installed and they can follow different guidelines for the placement of their files.

  2. Relative to the location specified by the package the actual translation files must be searched, based on the wishes of the user. I.e., for each language the user selects the program should be able to locate the appropriate file.

This is the functionality required by the specifications for gettext and this is also what the catgets functions are able to do. But there are some problems unresolved:

We can divide the configuration actions in two parts: the one is performed by the programmer, the other by the user. We will start with the functions the programmer can use since the user configuration will be based on this.

As the functions described in the last sections already mention separate sets of messages can be selected by a domain name. This is a simple string which should be unique for each program part with uses a separate domain. It is possible to use in one program arbitrary many domains at the same time. E.g., the GNU C Library itself uses a domain named libc while the program using the C Library could use a domain named foo. The important point is that at any time exactly one domain is active. This is controlled with the following function.

— Function: char * textdomain (const char *domainname)

The textdomain function sets the default domain, which is used in all future gettext calls, to domainname. Please note that dgettext and dcgettext calls are not influenced if the domainname parameter of these functions is not the null pointer.

Before the first call to textdomain the default domain is messages. This is the name specified in the specification of the gettext API. This name is as good as any other name. No program should ever really use a domain with this name since this can only lead to problems.

The function returns the value which is from now on taken as the default domain. If the system went out of memory the returned value is NULL and the global variable errno is set to ENOMEM. Despite the return value type being char * the return string must not be changed. It is allocated internally by the textdomain function.

If the domainname parameter is the null pointer no new default domain is set. Instead the currently selected default domain is returned.

If the domainname parameter is the empty string the default domain is reset to its initial value, the domain with the name messages. This possibility is questionable to use since the domain messages really never should be used.

— Function: char * bindtextdomain (const char *domainname, const char *dirname)

The bindtextdomain function can be used to specify the directory which contains the message catalogs for domain domainname for the different languages. To be correct, this is the directory where the hierarchy of directories is expected. Details are explained below.

For the programmer it is important to note that the translations which come with the program have be placed in a directory hierarchy starting at, say, /foo/bar. Then the program should make a bindtextdomain call to bind the domain for the current program to this directory. So it is made sure the catalogs are found. A correctly running program does not depend on the user setting an environment variable.

The bindtextdomain function can be used several times and if the domainname argument is different the previously bound domains will not be overwritten.

If the program which wish to use bindtextdomain at some point of time use the chdir function to change the current working directory it is important that the dirname strings ought to be an absolute pathname. Otherwise the addressed directory might vary with the time.

If the dirname parameter is the null pointer bindtextdomain returns the currently selected directory for the domain with the name domainname.

The bindtextdomain function returns a pointer to a string containing the name of the selected directory name. The string is allocated internally in the function and must not be changed by the user. If the system went out of core during the execution of bindtextdomain the return value is NULL and the global variable errno is set accordingly.

Next: , Previous: Locating gettext catalog, Up: Message catalogs with gettext
8.2.1.3 Additional functions for more complicated situations

The functions of the gettext family described so far (and all the catgets functions as well) have one problem in the real world which have been neglected completely in all existing approaches. What is meant here is the handling of plural forms.

Looking through Unix source code before the time anybody thought about internationalization (and, sadly, even afterwards) one can often find code similar to the following: 

        printf ("%d file%s deleted", n, n == 1 ? "" : "s");

After the first complaints from people internationalizing the code people either completely avoided formulations like this or used strings like "file(s)". Both look unnatural and should be avoided. First tries to solve the problem correctly looked like this: 

        if (n == 1)
          printf ("%d file deleted", n);
        else
          printf ("%d files deleted", n);

But this does not solve the problem. It helps languages where the plural form of a noun is not simply constructed by adding an `s' but that is all. Once again people fell into the trap of believing the rules their language is using are universal. But the handling of plural forms differs widely between the language families. There are two things we can differ between (and even inside language families);

The consequence of this is that application writers should not try to solve the problem in their code. This would be localization since it is only usable for certain, hardcoded language environments. Instead the extended gettext interface should be used.

These extra functions are taking instead of the one key string two strings and an numerical argument. The idea behind this is that using the numerical argument and the first string as a key, the implementation can select using rules specified by the translator the right plural form. The two string arguments then will be used to provide a return value in case no message catalog is found (similar to the normal gettext behavior). In this case the rules for Germanic language is used and it is assumed that the first string argument is the singular form, the second the plural form.

This has the consequence that programs without language catalogs can display the correct strings only if the program itself is written using a Germanic language. This is a limitation but since the GNU C library (as well as the GNU gettext package) are written as part of the GNU package and the coding standards for the GNU project require program being written in English, this solution nevertheless fulfills its purpose.

— Function: char * ngettext (const char *msgid1, const char *msgid2, unsigned long int n)

The ngettext function is similar to the gettext function as it finds the message catalogs in the same way. But it takes two extra arguments. The msgid1 parameter must contain the singular form of the string to be converted. It is also used as the key for the search in the catalog. The msgid2 parameter is the plural form. The parameter n is used to determine the plural form. If no message catalog is found msgid1 is returned if n == 1, otherwise msgid2.

An example for the us of this function is: 

            printf (ngettext ("%d file removed", "%d files removed", n), n);

Please note that the numeric value n has to be passed to the printf function as well. It is not sufficient to pass it only to ngettext.

— Function: char * dngettext (const char *domain, const char *msgid1, const char *msgid2, unsigned long int n)

The dngettext is similar to the dgettext function in the way the message catalog is selected. The difference is that it takes two extra parameter to provide the correct plural form. These two parameters are handled in the same way ngettext handles them.

— Function: char * dcngettext (const char *domain, const char *msgid1, const char *msgid2, unsigned long int n, int category)

The dcngettext is similar to the dcgettext function in the way the message catalog is selected. The difference is that it takes two extra parameter to provide the correct plural form. These two parameters are handled in the same way ngettext handles them.

The problem of plural forms

A description of the problem can be found at the beginning of the last section. Now there is the question how to solve it. Without the input of linguists (which was not available) it was not possible to determine whether there are only a few different forms in which plural forms are formed or whether the number can increase with every new supported language.

Therefore the solution implemented is to allow the translator to specify the rules of how to select the plural form. Since the formula varies with every language this is the only viable solution except for hardcoding the information in the code (which still would require the possibility of extensions to not prevent the use of new languages). The details are explained in the GNU gettext manual. Here only a bit of information is provided.

The information about the plural form selection has to be stored in the header entry (the one with the empty (msgid string). It looks like this: 

     Plural-Forms: nplurals=2; plural=n == 1 ? 0 : 1;

The nplurals value must be a decimal number which specifies how many different plural forms exist for this language. The string following plural is an expression which is using the C language syntax. Exceptions are that no negative number are allowed, numbers must be decimal, and the only variable allowed is n. This expression will be evaluated whenever one of the functions ngettext, dngettext, or dcngettext is called. The numeric value passed to these functions is then substituted for all uses of the variable n in the expression. The resulting value then must be greater or equal to zero and smaller than the value given as the value of nplurals.

The following rules are known at this point. The language with families are listed. But this does not necessarily mean the information can be generalized for the whole family (as can be easily seen in the table below).1

Only one form:
Some languages only require one single form. There is no distinction between the singular and plural form. An appropriate header entry would look like this: 
          Plural-Forms: nplurals=1; plural=0;

Languages with this property include:

Finno-Ugric family
Hungarian
Asian family
Japanese, Korean
Turkic/Altaic family
Turkish

Two forms, singular used for one only
This is the form used in most existing programs since it is what English is using. A header entry would look like this: 
          Plural-Forms: nplurals=2; plural=n != 1;

(Note: this uses the feature of C expressions that boolean expressions have to value zero or one.)

Languages with this property include:

Germanic family
Danish, Dutch, English, German, Norwegian, Swedish
Finno-Ugric family
Estonian, Finnish
Latin/Greek family
Greek
Semitic family
Hebrew
Romance family
Italian, Portuguese, Spanish
Artificial
Esperanto

Two forms, singular used for zero and one
Exceptional case in the language family. The header entry would be: 
          Plural-Forms: nplurals=2; plural=n>1;

Languages with this property include:

Romanic family
French, Brazilian Portuguese

Three forms, special case for zero
The header entry would be: 
          Plural-Forms: nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n != 0 ? 1 : 2;

Languages with this property include:

Baltic family
Latvian

Three forms, special cases for one and two
The header entry would be: 
          Plural-Forms: nplurals=3; plural=n==1 ? 0 : n==2 ? 1 : 2;

Languages with this property include:

Celtic
Gaeilge (Irish)

Three forms, special case for numbers ending in 1[2-9]
The header entry would look like this: 
          Plural-Forms: nplurals=3; \
              plural=n%10==1 && n%100!=11 ? 0 : \
                     n%10>=2 && (n%100<10 || n%100>=20) ? 1 : 2;

Languages with this property include:

Baltic family
Lithuanian

Three forms, special cases for numbers ending in 1 and 2, 3, 4, except those ending in 1[1-4]
The header entry would look like this: 
          Plural-Forms: nplurals=3; \
              plural=n%100/10==1 ? 2 : n%10==1 ? 0 : (n+9)%10>3 ? 2 : 1;

Languages with this property include:

Slavic family
Croatian, Czech, Russian, Ukrainian

Three forms, special cases for 1 and 2, 3, 4
The header entry would look like this: 
          Plural-Forms: nplurals=3; \
              plural=(n==1) ? 1 : (n>=2 && n<=4) ? 2 : 0;

Languages with this property include:

Slavic family
Slovak

Three forms, special case for one and some numbers ending in 2, 3, or 4
The header entry would look like this: 
          Plural-Forms: nplurals=3; \
              plural=n==1 ? 0 : \
                     n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

Languages with this property include:

Slavic family
Polish

Four forms, special case for one and all numbers ending in 02, 03, or 04
The header entry would look like this: 
          Plural-Forms: nplurals=4; \
              plural=n%100==1 ? 0 : n%100==2 ? 1 : n%100==3 || n%100==4 ? 2 : 3;

Languages with this property include:

Slavic family
Slovenian

Next: , Previous: Advanced gettext functions, Up: Message catalogs with gettext
8.2.1.4 How to specify the output character set gettext uses

gettext not only looks up a translation in a message catalog. It also converts the translation on the fly to the desired output character set. This is useful if the user is working in a different character set than the translator who created the message catalog, because it avoids distributing variants of message catalogs which differ only in the character set.

The output character set is, by default, the value of nl_langinfo (CODESET), which depends on the LC_CTYPE part of the current locale. But programs which store strings in a locale independent way (e.g. UTF-8) can request that gettext and related functions return the translations in that encoding, by use of the bind_textdomain_codeset function.

Note that the msgid argument to gettext is not subject to character set conversion. Also, when gettext does not find a translation for msgid, it returns msgid unchanged – independently of the current output character set. It is therefore recommended that all msgids be US-ASCII strings.

— Function: char * bind_textdomain_codeset (const char *domainname, const char *codeset)

The bind_textdomain_codeset function can be used to specify the output character set for message catalogs for domain domainname. The codeset argument must be a valid codeset name which can be used for the iconv_open function, or a null pointer.

If the codeset parameter is the null pointer, bind_textdomain_codeset returns the currently selected codeset for the domain with the name domainname. It returns NULL if no codeset has yet been selected.

The bind_textdomain_codeset function can be used several times. If used multiple times with the same domainname argument, the later call overrides the settings made by the earlier one.

The bind_textdomain_codeset function returns a pointer to a string containing the name of the selected codeset. The string is allocated internally in the function and must not be changed by the user. If the system went out of core during the execution of bind_textdomain_codeset, the return value is NULL and the global variable errno is set accordingly.

Next: , Previous: Charset conversion in gettext, Up: Message catalogs with gettext
8.2.1.5 How to use gettext in GUI programs

One place where the gettext functions, if used normally, have big problems is within programs with graphical user interfaces (GUIs). The problem is that many of the strings which have to be translated are very short. They have to appear in pull-down menus which restricts the length. But strings which are not containing entire sentences or at least large fragments of a sentence may appear in more than one situation in the program but might have different translations. This is especially true for the one-word strings which are frequently used in GUI programs.

As a consequence many people say that the gettext approach is wrong and instead catgets should be used which indeed does not have this problem. But there is a very simple and powerful method to handle these kind of problems with the gettext functions.

As as example consider the following fictional situation. A GUI program has a menu bar with the following entries: 

     +------------+------------+--------------------------------------+
     | File       | Printer    |                                      |
     +------------+------------+--------------------------------------+
     | Open     | | Select   |
     | New      | | Open     |
     +----------+ | Connect  |
                  +----------+

To have the strings File, Printer, Open, New, Select, and Connect translated there has to be at some point in the code a call to a function of the gettext family. But in two places the string passed into the function would be Open. The translations might not be the same and therefore we are in the dilemma described above.

One solution to this problem is to artificially enlengthen the strings to make them unambiguous. But what would the program do if no translation is available? The enlengthened string is not what should be printed. So we should use a little bit modified version of the functions.

To enlengthen the strings a uniform method should be used. E.g., in the example above the strings could be chosen as 

     Menu|File
     Menu|Printer
     Menu|File|Open
     Menu|File|New
     Menu|Printer|Select
     Menu|Printer|Open
     Menu|Printer|Connect

Now all the strings are different and if now instead of gettext the following little wrapper function is used, everything works just fine: 

       char *
       sgettext (const char *msgid)
       {
         char *msgval = gettext (msgid);
         if (msgval == msgid)
           msgval = strrchr (msgid, '|') + 1;
         return msgval;
       }

What this little function does is to recognize the case when no translation is available. This can be done very efficiently by a pointer comparison since the return value is the input value. If there is no translation we know that the input string is in the format we used for the Menu entries and therefore contains a | character. We simply search for the last occurrence of this character and return a pointer to the character following it. That's it!

If one now consistently uses the enlengthened string form and replaces the gettext calls with calls to sgettext (this is normally limited to very few places in the GUI implementation) then it is possible to produce a program which can be internationalized.

With advanced compilers (such as GNU C) one can write the sgettext functions as an inline function or as a macro like this: 

     #define sgettext(msgid) \
       ({ const char *__msgid = (msgid);            \
          char *__msgstr = gettext (__msgid);       \
          if (__msgval == __msgid)                  \
            __msgval = strrchr (__msgid, '|') + 1;  \
          __msgval; })

The other gettext functions (dgettext, dcgettext and the ngettext equivalents) can and should have corresponding functions as well which look almost identical, except for the parameters and the call to the underlying function.

Now there is of course the question why such functions do not exist in the GNU C library? There are two parts of the answer to this question.

There is only one more comment to make left. The wrapper function above require that the translations strings are not enlengthened themselves. This is only logical. There is no need to disambiguate the strings (since they are never used as keys for a search) and one also saves quite some memory and disk space by doing this.

Previous: GUI program problems, Up: Message catalogs with gettext
8.2.1.6 User influence on gettext

The last sections described what the programmer can do to internationalize the messages of the program. But it is finally up to the user to select the message s/he wants to see. S/He must understand them.

The POSIX locale model uses the environment variables LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC, and LC_TIME to select the locale which is to be used. This way the user can influence lots of functions. As we mentioned above the gettext functions also take advantage of this.

To understand how this happens it is necessary to take a look at the various components of the filename which gets computed to locate a message catalog. It is composed as follows: 

     dir_name/locale/LC_category/domain_name.mo

The default value for dir_name is system specific. It is computed from the value given as the prefix while configuring the C library. This value normally is /usr or /. For the former the complete dir_name is: 

     /usr/share/locale

We can use /usr/share since the .mo files containing the message catalogs are system independent, so all systems can use the same files. If the program executed the bindtextdomain function for the message domain that is currently handled, the dir_name component is exactly the value which was given to the function as the second parameter. I.e., bindtextdomain allows overwriting the only system dependent and fixed value to make it possible to address files anywhere in the filesystem.

The category is the name of the locale category which was selected in the program code. For gettext and dgettext this is always LC_MESSAGES, for dcgettext this is selected by the value of the third parameter. As said above it should be avoided to ever use a category other than LC_MESSAGES.

The locale component is computed based on the category used. Just like for the setlocale function here comes the user selection into the play. Some environment variables are examined in a fixed order and the first environment variable set determines the return value of the lookup process. In detail, for the category LC_xxx the following variables in this order are examined:

LANGUAGE
LC_ALL
LC_xxx
LANG

This looks very familiar. With the exception of the LANGUAGE environment variable this is exactly the lookup order the setlocale function uses. But why introducing the LANGUAGE variable?

The reason is that the syntax of the values these variables can have is different to what is expected by the setlocale function. If we would set LC_ALL to a value following the extended syntax that would mean the setlocale function will never be able to use the value of this variable as well. An additional variable removes this problem plus we can select the language independently of the locale setting which sometimes is useful.

While for the LC_xxx variables the value should consist of exactly one specification of a locale the LANGUAGE variable's value can consist of a colon separated list of locale names. The attentive reader will realize that this is the way we manage to implement one of our additional demands above: we want to be able to specify an ordered list of language.

Back to the constructed filename we have only one component missing. The domain_name part is the name which was either registered using the textdomain function or which was given to dgettext or dcgettext as the first parameter. Now it becomes obvious that a good choice for the domain name in the program code is a string which is closely related to the program/package name. E.g., for the GNU C Library the domain name is libc.

A limit piece of example code should show how the programmer is supposed to work: 

     {
       setlocale (LC_ALL, "");
       textdomain ("test-package");
       bindtextdomain ("test-package", "/usr/local/share/locale");
       puts (gettext ("Hello, world!"));
     }

At the program start the default domain is messages, and the default locale is "C". The setlocale call sets the locale according to the user's environment variables; remember that correct functioning of gettext relies on the correct setting of the LC_MESSAGES locale (for looking up the message catalog) and of the LC_CTYPE locale (for the character set conversion). The textdomain call changes the default domain to test-package. The bindtextdomain call specifies that the message catalogs for the domain test-package can be found below the directory /usr/local/share/locale.

If now the user set in her/his environment the variable LANGUAGE to de the gettext function will try to use the translations from the file 

     /usr/local/share/locale/de/LC_MESSAGES/test-package.mo

From the above descriptions it should be clear which component of this filename is determined by which source.

In the above example we assumed that the LANGUAGE environment variable to de. This might be an appropriate selection but what happens if the user wants to use LC_ALL because of the wider usability and here the required value is de_DE.ISO-8859-1? We already mentioned above that a situation like this is not infrequent. E.g., a person might prefer reading a dialect and if this is not available fall back on the standard language.

The gettext functions know about situations like this and can handle them gracefully. The functions recognize the format of the value of the environment variable. It can split the value is different pieces and by leaving out the only or the other part it can construct new values. This happens of course in a predictable way. To understand this one must know the format of the environment variable value. There is one more or less standardized form, originally from the X/Open specification:

language[_territory[.codeset]][@modifier]

Less specific locale names will be stripped of in the order of the following list:

  1. codeset
  2. normalized codeset
  3. territory
  4. modifier

The language field will never be dropped for obvious reasons.

The only new thing is the normalized codeset entry. This is another goodie which is introduced to help reducing the chaos which derives from the inability of the people to standardize the names of character sets. Instead of ISO-8859-1 one can often see 8859-1, 88591, iso8859-1, or iso_8859-1. The normalized codeset value is generated from the user-provided character set name by applying the following rules:

  1. Remove all characters beside numbers and letters.
  2. Fold letters to lowercase.
  3. If the same only contains digits prepend the string "iso".

So all of the above name will be normalized to iso88591. This allows the program user much more freely choosing the locale name.

Even this extended functionality still does not help to solve the problem that completely different names can be used to denote the same locale (e.g., de and german). To be of help in this situation the locale implementation and also the gettext functions know about aliases.

The file /usr/share/locale/locale.alias (replace /usr with whatever prefix you used for configuring the C library) contains a mapping of alternative names to more regular names. The system manager is free to add new entries to fill her/his own needs. The selected locale from the environment is compared with the entries in the first column of this file ignoring the case. If they match the value of the second column is used instead for the further handling.

In the description of the format of the environment variables we already mentioned the character set as a factor in the selection of the message catalog. In fact, only catalogs which contain text written using the character set of the system/program can be used (directly; there will come a solution for this some day). This means for the user that s/he will always have to take care for this. If in the collection of the message catalogs there are files for the same language but coded using different character sets the user has to be careful.

Previous: Message catalogs with gettext, Up: The Uniforum approach

8.2.2 Programs to handle message catalogs for gettext

The GNU C Library does not contain the source code for the programs to handle message catalogs for the gettext functions. As part of the GNU project the GNU gettext package contains everything the developer needs. The functionality provided by the tools in this package by far exceeds the abilities of the gencat program described above for the catgets functions.

There is a program msgfmt which is the equivalent program to the gencat program. It generates from the human-readable and -editable form of the message catalog a binary file which can be used by the gettext functions. But there are several more programs available.

The xgettext program can be used to automatically extract the translatable messages from a source file. I.e., the programmer need not take care for the translations and the list of messages which have to be translated. S/He will simply wrap the translatable string in calls to gettext et.al and the rest will be done by xgettext. This program has a lot of option which help to customize the output or do help to understand the input better.

Other programs help to manage development cycle when new messages appear in the source files or when a new translation of the messages appear. Here it should only be noted that using all the tools in GNU gettext it is possible to completely automate the handling of message catalog. Beside marking the translatable string in the source code and generating the translations the developers do not have anything to do themselves.

Next: , Previous: Message Translation, Up: Top

9 Searching and Sorting

This chapter describes functions for searching and sorting arrays of arbitrary objects. You pass the appropriate comparison function to be applied as an argument, along with the size of the objects in the array and the total number of elements.

Next: , Up: Searching and Sorting

9.1 Defining the Comparison Function

In order to use the sorted array library functions, you have to describe how to compare the elements of the array.

To do this, you supply a comparison function to compare two elements of the array. The library will call this function, passing as arguments pointers to two array elements to be compared. Your comparison function should return a value the way strcmp (see String/Array Comparison) does: negative if the first argument is “less” than the second, zero if they are “equal”, and positive if the first argument is “greater”.

Here is an example of a comparison function which works with an array of numbers of type double: 

     int
     compare_doubles (const void *a, const void *b)
     {
       const double *da = (const double *) a;
       const double *db = (const double *) b;
     
       return (*da > *db) - (*da < *db);
     }

The header file stdlib.h defines a name for the data type of comparison functions. This type is a GNU extension. 

     int comparison_fn_t (const void *, const void *);

Next: , Previous: Comparison Functions, Up: Searching and Sorting

9.2 Array Search Function

Generally searching for a specific element in an array means that potentially all elements must be checked. The GNU C library contains functions to perform linear search. The prototypes for the following two functions can be found in search.h.

— Function: void * lfind (const void *key, void *base, size_t *nmemb, size_t size, comparison_fn_t compar)

The lfind function searches in the array with *nmemb elements of size bytes pointed to by base for an element which matches the one pointed to by key. The function pointed to by compar is used decide whether two elements match.

The return value is a pointer to the matching element in the array starting at base if it is found. If no matching element is available NULL is returned.

The mean runtime of this function is *nmemb/2. This function should only be used if elements often get added to or deleted from the array in which case it might not be useful to sort the array before searching.

— Function: void * lsearch (const void *key, void *base, size_t *nmemb, size_t size, comparison_fn_t compar)

The lsearch function is similar to the lfind function. It searches the given array for an element and returns it if found. The difference is that if no matching element is found the lsearch function adds the object pointed to by key (with a size of size bytes) at the end of the array and it increments the value of *nmemb to reflect this addition.

This means for the caller that if it is not sure that the array contains the element one is searching for the memory allocated for the array starting at base must have room for at least size more bytes. If one is sure the element is in the array it is better to use lfind so having more room in the array is always necessary when calling lsearch.

To search a sorted array for an element matching the key, use the bsearch function. The prototype for this function is in the header file stdlib.h.

— Function: void * bsearch (const void *key, const void *array, size_t count, size_t size, comparison_fn_t compare)

The bsearch function searches the sorted array array for an object that is equivalent to key. The array contains count elements, each of which is of size size bytes.

The compare function is used to perform the comparison. This function is called with two pointer arguments and should return an integer less than, equal to, or greater than zero corresponding to whether its first argument is considered less than, equal to, or greater than its second argument. The elements of the array must already be sorted in ascending order according to this comparison function.

The return value is a pointer to the matching array element, or a null pointer if no match is found. If the array contains more than one element that matches, the one that is returned is unspecified.

This function derives its name from the fact that it is implemented using the binary search algorithm.

Next: , Previous: Array Search Function, Up: Searching and Sorting

9.3 Array Sort Function

To sort an array using an arbitrary comparison function, use the qsort function. The prototype for this function is in stdlib.h.

— Function: void qsort (void *array, size_t count, size_t size, comparison_fn_t compare)

The qsort function sorts the array array. The array contains count elements, each of which is of size size.

The compare function is used to perform the comparison on the array elements. This function is called with two pointer arguments and should return an integer less than, equal to, or greater than zero corresponding to whether its first argument is considered less than, equal to, or greater than its second argument.

Warning: If two objects compare as equal, their order after sorting is unpredictable. That is to say, the sorting is not stable. This can make a difference when the comparison considers only part of the elements. Two elements with the same sort key may differ in other respects.

If you want the effect of a stable sort, you can get this result by writing the comparison function so that, lacking other reason distinguish between two elements, it compares them by their addresses. Note that doing this may make the sorting algorithm less efficient, so do it only if necessary.

Here is a simple example of sorting an array of doubles in numerical order, using the comparison function defined above (see Comparison Functions): 

          {
            double *array;
            int size;
            ...
            qsort (array, size, sizeof (double), compare_doubles);
          }

The qsort function derives its name from the fact that it was originally implemented using the “quick sort” algorithm.

The implementation of qsort in this library might not be an in-place sort and might thereby use an extra amount of memory to store the array.

Next: , Previous: Array Sort Function, Up: Searching and Sorting

9.4 Searching and Sorting Example

Here is an example showing the use of qsort and bsearch with an array of structures. The objects in the array are sorted by comparing their name fields with the strcmp function. Then, we can look up individual objects based on their names. 

     #include <stdlib.h>
     #include <stdio.h>
     #include <string.h>
     
     /* Define an array of critters to sort. */
     
     struct critter
       {
         const char *name;
         const char *species;
       };
     
     struct critter muppets[] =
       {
         {"Kermit", "frog"},
         {"Piggy", "pig"},
         {"Gonzo", "whatever"},
         {"Fozzie", "bear"},
         {"Sam", "eagle"},
         {"Robin", "frog"},
         {"Animal", "animal"},
         {"Camilla", "chicken"},
         {"Sweetums", "monster"},
         {"Dr. Strangepork", "pig"},
         {"Link Hogthrob", "pig"},
         {"Zoot", "human"},
         {"Dr. Bunsen Honeydew", "human"},
         {"Beaker", "human"},
         {"Swedish Chef", "human"}
       };
     
     int count = sizeof (muppets) / sizeof (struct critter);
     
     
     
     /* This is the comparison function used for sorting and searching. */
     
     int
     critter_cmp (const struct critter *c1, const struct critter *c2)
     {
       return strcmp (c1->name, c2->name);
     }
     
     
     /* Print information about a critter. */
     
     void
     print_critter (const struct critter *c)
     {
       printf ("%s, the %s\n", c->name, c->species);
     }
     
     
     /* Do the lookup into the sorted array. */
     
     void
     find_critter (const char *name)
     {
       struct critter target, *result;
       target.name = name;
       result = bsearch (&target, muppets, count, sizeof (struct critter),
                         critter_cmp);
       if (result)
         print_critter (result);
       else
         printf ("Couldn't find %s.\n", name);
     }
     
     /* Main program. */
     
     int
     main (void)
     {
       int i;
     
       for (i = 0; i < count; i++)
         print_critter (&muppets[i]);
       printf ("\n");
     
       qsort (muppets, count, sizeof (struct critter), critter_cmp);
     
       for (i = 0; i < count; i++)
         print_critter (&muppets[i]);
       printf ("\n");
     
       find_critter ("Kermit");
       find_critter ("Gonzo");
       find_critter ("Janice");
     
       return 0;
     }

The output from this program looks like: 

     Kermit, the frog
     Piggy, the pig
     Gonzo, the whatever
     Fozzie, the bear
     Sam, the eagle
     Robin, the frog
     Animal, the animal
     Camilla, the chicken
     Sweetums, the monster
     Dr. Strangepork, the pig
     Link Hogthrob, the pig
     Zoot, the human
     Dr. Bunsen Honeydew, the human
     Beaker, the human
     Swedish Chef, the human
     
     Animal, the animal
     Beaker, the human
     Camilla, the chicken
     Dr. Bunsen Honeydew, the human
     Dr. Strangepork, the pig
     Fozzie, the bear
     Gonzo, the whatever
     Kermit, the frog
     Link Hogthrob, the pig
     Piggy, the pig
     Robin, the frog
     Sam, the eagle
     Swedish Chef, the human
     Sweetums, the monster
     Zoot, the human
     
     Kermit, the frog
     Gonzo, the whatever
     Couldn't find Janice.

Next: , Previous: Search/Sort Example, Up: Searching and Sorting

9.5 The hsearch function.

The functions mentioned so far in this chapter are for searching in a sorted or unsorted array. There are other methods to organize information which later should be searched. The costs of insert, delete and search differ. One possible implementation is using hashing tables. The following functions are declared in the header file search.h.

— Function: int hcreate (size_t nel)

The hcreate function creates a hashing table which can contain at least nel elements. There is no possibility to grow this table so it is necessary to choose the value for nel wisely. The method used to implement this function might make it necessary to make the number of elements in the hashing table larger than the expected maximal number of elements. Hashing tables usually work inefficiently if they are filled 80% or more. The constant access time guaranteed by hashing can only be achieved if few collisions exist. See Knuth's “The Art of Computer Programming, Part 3: Searching and Sorting” for more information.

The weakest aspect of this function is that there can be at most one hashing table used through the whole program. The table is allocated in local memory out of control of the programmer. As an extension the GNU C library provides an additional set of functions with an reentrant interface which provide a similar interface but which allow to keep arbitrarily many hashing tables.

It is possible to use more than one hashing table in the program run if the former table is first destroyed by a call to hdestroy.

The function returns a non-zero value if successful. If it return zero something went wrong. This could either mean there is already a hashing table in use or the program runs out of memory.

— Function: void hdestroy (void)

The hdestroy function can be used to free all the resources allocated in a previous call of hcreate. After a call to this function it is again possible to call hcreate and allocate a new table with possibly different size.

It is important to remember that the elements contained in the hashing table at the time hdestroy is called are not freed by this function. It is the responsibility of the program code to free those strings (if necessary at all). Freeing all the element memory is not possible without extra, separately kept information since there is no function to iterate through all available elements in the hashing table. If it is really necessary to free a table and all elements the programmer has to keep a list of all table elements and before calling hdestroy s/he has to free all element's data using this list. This is a very unpleasant mechanism and it also shows that this kind of hashing tables is mainly meant for tables which are created once and used until the end of the program run.

Entries of the hashing table and keys for the search are defined using this type:

— Data type: struct ENTRY

Both elements of this structure are pointers to zero-terminated strings. This is a limiting restriction of the functionality of the hsearch functions. They can only be used for data sets which use the NUL character always and solely to terminate the records. It is not possible to handle general binary data.

char *key
Pointer to a zero-terminated string of characters describing the key for the search or the element in the hashing table.
char *data
Pointer to a zero-terminated string of characters describing the data. If the functions will be called only for searching an existing entry this element might stay undefined since it is not used.

— Function: ENTRY * hsearch (ENTRY item, ACTION action)

To search in a hashing table created using hcreate the hsearch function must be used. This function can perform simple search for an element (if action has the FIND) or it can alternatively insert the key element into the hashing table. Entries are never replaced.

The key is denoted by a pointer to an object of type ENTRY. For locating the corresponding position in the hashing table only the key element of the structure is used.

If an entry with matching key is found the action parameter is irrelevant. The found entry is returned. If no matching entry is found and the action parameter has the value FIND the function returns a NULL pointer. If no entry is found and the action parameter has the value ENTER a new entry is added to the hashing table which is initialized with the parameter item. A pointer to the newly added entry is returned.

As mentioned before the hashing table used by the functions described so far is global and there can be at any time at most one hashing table in the program. A solution is to use the following functions which are a GNU extension. All have in common that they operate on a hashing table which is described by the content of an object of the type struct hsearch_data. This type should be treated as opaque, none of its members should be changed directly.

— Function: int hcreate_r (size_t nel, struct hsearch_data *htab)

The hcreate_r function initializes the object pointed to by htab to contain a hashing table with at least nel elements. So this function is equivalent to the hcreate function except that the initialized data structure is controlled by the user.

This allows having more than one hashing table at one time. The memory necessary for the struct hsearch_data object can be allocated dynamically. It must be initialized with zero before calling this function.

The return value is non-zero if the operation was successful. If the return value is zero, something went wrong, which probably means the programs ran out of memory.

— Function: void hdestroy_r (struct hsearch_data *htab)

The hdestroy_r function frees all resources allocated by the hcreate_r function for this very same object htab. As for hdestroy it is the programs responsibility to free the strings for the elements of the table.

— Function: int hsearch_r (ENTRY item, ACTION action, ENTRY **retval, struct hsearch_data *htab)

The hsearch_r function is equivalent to hsearch. The meaning of the first two arguments is identical. But instead of operating on a single global hashing table the function works on the table described by the object pointed to by htab (which is initialized by a call to hcreate_r).

Another difference to hcreate is that the pointer to the found entry in the table is not the return value of the functions. It is returned by storing it in a pointer variables pointed to by the retval parameter. The return value of the function is an integer value indicating success if it is non-zero and failure if it is zero. In the latter case the global variable errno signals the reason for the failure.

ENOMEM
The table is filled and hsearch_r was called with an so far unknown key and action set to ENTER.
ESRCH
The action parameter is FIND and no corresponding element is found in the table.

Previous: Hash Search Function, Up: Searching and Sorting

9.6 The tsearch function.

Another common form to organize data for efficient search is to use trees. The tsearch function family provides a nice interface to functions to organize possibly large amounts of data by providing a mean access time proportional to the logarithm of the number of elements. The GNU C library implementation even guarantees that this bound is never exceeded even for input data which cause problems for simple binary tree implementations.

The functions described in the chapter are all described in the System V and X/Open specifications and are therefore quite portable.

In contrast to the hsearch functions the tsearch functions can be used with arbitrary data and not only zero-terminated strings.

The tsearch functions have the advantage that no function to initialize data structures is necessary. A simple pointer of type void * initialized to NULL is a valid tree and can be extended or searched. The prototypes for these functions can be found in the header file search.h.

— Function: void * tsearch (const void *key, void **rootp, comparison_fn_t compar)

The tsearch function searches in the tree pointed to by *rootp for an element matching key. The function pointed to by compar is used to determine whether two elements match. See Comparison Functions, for a specification of the functions which can be used for the compar parameter.

If the tree does not contain a matching entry the key value will be added to the tree. tsearch does not make a copy of the object pointed to by key (how could it since the size is unknown). Instead it adds a reference to this object which means the object must be available as long as the tree data structure is used.

The tree is represented by a pointer to a pointer since it is sometimes necessary to change the root node of the tree. So it must not be assumed that the variable pointed to by rootp has the same value after the call. This also shows that it is not safe to call the tsearch function more than once at the same time using the same tree. It is no problem to run it more than once at a time on different trees.

The return value is a pointer to the matching element in the tree. If a new element was created the pointer points to the new data (which is in fact key). If an entry had to be created and the program ran out of space NULL is returned.

— Function: void * tfind (const void *key, void *const *rootp, comparison_fn_t compar)

The tfind function is similar to the tsearch function. It locates an element matching the one pointed to by key and returns a pointer to this element. But if no matching element is available no new element is entered (note that the rootp parameter points to a constant pointer). Instead the function returns NULL.

Another advantage of the tsearch function in contrast to the hsearch functions is that there is an easy way to remove elements.

— Function: void * tdelete (const void *key, void **rootp, comparison_fn_t compar)

To remove a specific element matching key from the tree tdelete can be used. It locates the matching element using the same method as tfind. The corresponding element is then removed and a pointer to the parent of the deleted node is returned by the function. If there is no matching entry in the tree nothing can be deleted and the function returns NULL. If the root of the tree is deleted tdelete returns some unspecified value not equal to NULL.

— Function: void tdestroy (void *vroot, __free_fn_t freefct)

If the complete search tree has to be removed one can use tdestroy. It frees all resources allocated by the tsearch function to generate the tree pointed to by vroot.

For the data in each tree node the function freefct is called. The pointer to the data is passed as the argument to the function. If no such work is necessary freefct must point to a function doing nothing. It is called in any case.

This function is a GNU extension and not covered by the System V or X/Open specifications.

In addition to the function to create and destroy the tree data structure, there is another function which allows you to apply a function to all elements of the tree. The function must have this type: 

     void __action_fn_t (const void *nodep, VISIT value, int level);

The nodep is the data value of the current node (once given as the key argument to tsearch). level is a numeric value which corresponds to the depth of the current node in the tree. The root node has the depth 0 and its children have a depth of 1 and so on. The VISIT type is an enumeration type.

— Data Type: VISIT

The VISIT value indicates the status of the current node in the tree and how the function is called. The status of a node is either `leaf' or `internal node'. For each leaf node the function is called exactly once, for each internal node it is called three times: before the first child is processed, after the first child is processed and after both children are processed. This makes it possible to handle all three methods of tree traversal (or even a combination of them).

preorder
The current node is an internal node and the function is called before the first child was processed.
postorder
The current node is an internal node and the function is called after the first child was processed.
endorder
The current node is an internal node and the function is called after the second child was processed.
leaf
The current node is a leaf.

— Function: void twalk (const void *root, __action_fn_t action)

For each node in the tree with a node pointed to by root, the twalk function calls the function provided by the parameter action. For leaf nodes the function is called exactly once with value set to leaf. For internal nodes the function is called three times, setting the value parameter or action to the appropriate value. The level argument for the action function is computed while descending the tree with increasing the value by one for the descend to a child, starting with the value 0 for the root node.

Since the functions used for the action parameter to twalk must not modify the tree data, it is safe to run twalk in more than one thread at the same time, working on the same tree. It is also safe to call tfind in parallel. Functions which modify the tree must not be used, otherwise the behavior is undefined.

Next: , Previous: Searching and Sorting, Up: Top

10 Pattern Matching

The GNU C Library provides pattern matching facilities for two kinds of patterns: regular expressions and file-name wildcards. The library also provides a facility for expanding variable and command references and parsing text into words in the way the shell does.

Next: , Up: Pattern Matching

10.1 Wildcard Matching

This section describes how to match a wildcard pattern against a particular string. The result is a yes or no answer: does the string fit the pattern or not. The symbols described here are all declared in fnmatch.h.

— Function: int fnmatch (const char *pattern, const char *string, int flags)

This function tests whether the string string matches the pattern pattern. It returns 0 if they do match; otherwise, it returns the nonzero value FNM_NOMATCH. The arguments pattern and string are both strings.

The argument flags is a combination of flag bits that alter the details of matching. See below for a list of the defined flags.

In the GNU C Library, fnmatch cannot experience an “error”—it always returns an answer for whether the match succeeds. However, other implementations of fnmatch might sometimes report “errors”. They would do so by returning nonzero values that are not equal to FNM_NOMATCH.

These are the available flags for the flags argument:

FNM_FILE_NAME
Treat the `/' character specially, for matching file names. If this flag is set, wildcard constructs in pattern cannot match `/' in string. Thus, the only way to match `/' is with an explicit `/' in pattern.
FNM_PATHNAME
This is an alias for FNM_FILE_NAME; it comes from POSIX.2. We don't recommend this name because we don't use the term “pathname” for file names.
FNM_PERIOD
Treat the `.' character specially if it appears at the beginning of string. If this flag is set, wildcard constructs in pattern cannot match `.' as the first character of string.

If you set both FNM_PERIOD and FNM_FILE_NAME, then the special treatment applies to `.' following `/' as well as to `.' at the beginning of string. (The shell uses the FNM_PERIOD and FNM_FILE_NAME flags together for matching file names.)

FNM_NOESCAPE
Don't treat the `\' character specially in patterns. Normally, `\' quotes the following character, turning off its special meaning (if any) so that it matches only itself. When quoting is enabled, the pattern `\?' matches only the string `?', because the question mark in the pattern acts like an ordinary character.

If you use FNM_NOESCAPE, then `\' is an ordinary character.

FNM_LEADING_DIR
Ignore a trailing sequence of characters starting with a `/' in string; that is to say, test whether string starts with a directory name that pattern matches.

If this flag is set, either `foo*' or `foobar' as a pattern would match the string `foobar/frobozz'.

FNM_CASEFOLD
Ignore case in comparing string to pattern.
FNM_EXTMATCH
Recognize beside the normal patterns also the extended patterns introduced in ksh. The patterns are written in the form explained in the following table where pattern-list is a | separated list of patterns.
?(pattern-list)
The pattern matches if zero or one occurrences of any of the patterns in the pattern-list allow matching the input string.
*(pattern-list)
The pattern matches if zero or more occurrences of any of the patterns in the pattern-list allow matching the input string.
+(pattern-list)
The pattern matches if one or more occurrences of any of the patterns in the pattern-list allow matching the input string.
@(pattern-list)
The pattern matches if exactly one occurrence of any of the patterns in the pattern-list allows matching the input string.
!(pattern-list)
The pattern matches if the input string cannot be matched with any of the patterns in the pattern-list.

Next: , Previous: Wildcard Matching, Up: Pattern Matching

10.2 Globbing

The archetypal use of wildcards is for matching against the files in a directory, and making a list of all the matches. This is called globbing.

You could do this using fnmatch, by reading the directory entries one by one and testing each one with fnmatch. But that would be slow (and complex, since you would have to handle subdirectories by hand).

The library provides a function glob to make this particular use of wildcards convenient. glob and the other symbols in this section are declared in glob.h.

Next: , Up: Globbing

10.2.1 Calling glob

The result of globbing is a vector of file names (strings). To return this vector, glob uses a special data type, glob_t, which is a structure. You pass glob the address of the structure, and it fills in the structure's fields to tell you about the results.

— Data Type: glob_t

This data type holds a pointer to a word vector. More precisely, it records both the address of the word vector and its size. The GNU implementation contains some more fields which are non-standard extensions.

gl_pathc
The number of elements in the vector, excluding the initial null entries if the GLOB_DOOFFS flag is used (see gl_offs below).
gl_pathv
The address of the vector. This field has type char **.
gl_offs
The offset of the first real element of the vector, from its nominal address in the gl_pathv field. Unlike the other fields, this is always an input to glob, rather than an output from it.

If you use a nonzero offset, then that many elements at the beginning of the vector are left empty. (The glob function fills them with null pointers.)

The gl_offs field is meaningful only if you use the GLOB_DOOFFS flag. Otherwise, the offset is always zero regardless of what is in this field, and the first real element comes at the beginning of the vector.

gl_closedir
The address of an alternative implementation of the closedir function. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is void (*) (void *).

This is a GNU extension.

gl_readdir
The address of an alternative implementation of the readdir function used to read the contents of a directory. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is struct dirent *(*) (void *).

This is a GNU extension.

gl_opendir
The address of an alternative implementation of the opendir function. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is void *(*) (const char *).

This is a GNU extension.

gl_stat
The address of an alternative implementation of the stat function to get information about an object in the filesystem. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is int (*) (const char *, struct stat *).

This is a GNU extension.

gl_lstat
The address of an alternative implementation of the lstat function to get information about an object in the filesystems, not following symbolic links. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is int (*) (const char *, struct stat *).

This is a GNU extension.

For use in the glob64 function glob.h contains another definition for a very similar type. glob64_t differs from glob_t only in the types of the members gl_readdir, gl_stat, and gl_lstat.

— Data Type: glob64_t

This data type holds a pointer to a word vector. More precisely, it records both the address of the word vector and its size. The GNU implementation contains some more fields which are non-standard extensions.

gl_pathc
The number of elements in the vector, excluding the initial null entries if the GLOB_DOOFFS flag is used (see gl_offs below).
gl_pathv
The address of the vector. This field has type char **.
gl_offs
The offset of the first real element of the vector, from its nominal address in the gl_pathv field. Unlike the other fields, this is always an input to glob, rather than an output from it.

If you use a nonzero offset, then that many elements at the beginning of the vector are left empty. (The glob function fills them with null pointers.)

The gl_offs field is meaningful only if you use the GLOB_DOOFFS flag. Otherwise, the offset is always zero regardless of what is in this field, and the first real element comes at the beginning of the vector.

gl_closedir
The address of an alternative implementation of the closedir function. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is void (*) (void *).

This is a GNU extension.

gl_readdir
The address of an alternative implementation of the readdir64 function used to read the contents of a directory. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is struct dirent64 *(*) (void *).

This is a GNU extension.

gl_opendir
The address of an alternative implementation of the opendir function. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is void *(*) (const char *).

This is a GNU extension.

gl_stat
The address of an alternative implementation of the stat64 function to get information about an object in the filesystem. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is int (*) (const char *, struct stat64 *).

This is a GNU extension.

gl_lstat
The address of an alternative implementation of the lstat64 function to get information about an object in the filesystems, not following symbolic links. It is used if the GLOB_ALTDIRFUNC bit is set in the flag parameter. The type of this field is int (*) (const char *, struct stat64 *).

This is a GNU extension.

— Function: int glob (const char *pattern, int flags, int (*errfunc) (const char *filename, int error-code), glob_t *vector-ptr)

The function glob does globbing using the pattern pattern in the current directory. It puts the result in a newly allocated vector, and stores the size and address of this vector into *vector-ptr. The argument flags is a combination of bit flags; see Flags for Globbing, for details of the flags.

The result of globbing is a sequence of file names. The function glob allocates a string for each resulting word, then allocates a vector of type char ** to store the addresses of these strings. The last element of the vector is a null pointer. This vector is called the word vector.

To return this vector, glob stores both its address and its length (number of elements, not counting the terminating null pointer) into *vector-ptr.

Normally, glob sorts the file names alphabetically before returning them. You can turn this off with the flag GLOB_NOSORT if you want to get the information as fast as possible. Usually it's a good idea to let glob sort them—if you process the files in alphabetical order, the users will have a feel for the rate of progress that your application is making.

If glob succeeds, it returns 0. Otherwise, it returns one of these error codes:

GLOB_ABORTED
There was an error opening a directory, and you used the flag GLOB_ERR or your specified errfunc returned a nonzero value. for an explanation of the GLOB_ERR flag and errfunc.
GLOB_NOMATCH
The pattern didn't match any existing files. If you use the GLOB_NOCHECK flag, then you never get this error code, because that flag tells glob to pretend that the pattern matched at least one file.
GLOB_NOSPACE
It was impossible to allocate memory to hold the result.

In the event of an error, glob stores information in *vector-ptr about all the matches it has found so far.

It is important to notice that the glob function will not fail if it encounters directories or files which cannot be handled without the LFS interfaces. The implementation of glob is supposed to use these functions internally. This at least is the assumptions made by the Unix standard. The GNU extension of allowing the user to provide own directory handling and stat functions complicates things a bit. If these callback functions are used and a large file or directory is encountered glob can fail.

— Function: int glob64 (const char *pattern, int flags, int (*errfunc) (const char *filename, int error-code), glob64_t *vector-ptr)

The glob64 function was added as part of the Large File Summit extensions but is not part of the original LFS proposal. The reason for this is simple: it is not necessary. The necessity for a glob64 function is added by the extensions of the GNU glob implementation which allows the user to provide own directory handling and stat functions. The readdir and stat functions do depend on the choice of _FILE_OFFSET_BITS since the definition of the types struct dirent and struct stat will change depending on the choice.

Beside this difference the glob64 works just like glob in all aspects.

This function is a GNU extension.

Next: , Previous: Calling Glob, Up: Globbing

10.2.2 Flags for Globbing

This section describes the flags that you can specify in the flags argument to glob. Choose the flags you want, and combine them with the C bitwise OR operator |.

GLOB_APPEND
Append the words from this expansion to the vector of words produced by previous calls to glob. This way you can effectively expand several words as if they were concatenated with spaces between them.

In order for appending to work, you must not modify the contents of the word vector structure between calls to glob. And, if you set GLOB_DOOFFS in the first call to glob, you must also set it when you append to the results.

Note that the pointer stored in gl_pathv may no longer be valid after you call glob the second time, because glob might have relocated the vector. So always fetch gl_pathv from the glob_t structure after each glob call; never save the pointer across calls.

GLOB_DOOFFS
Leave blank slots at the beginning of the vector of words. The gl_offs field says how many slots to leave. The blank slots contain null pointers.
GLOB_ERR
Give up right away and report an error if there is any difficulty reading the directories that must be read in order to expand pattern fully. Such difficulties might include a directory in which you don't have the requisite access. Normally, glob tries its best to keep on going despite any errors, reading whatever directories it can.

You can exercise even more control than this by specifying an error-handler function errfunc when you call glob. If errfunc is not a null pointer, then glob doesn't give up right away when it can't read a directory; instead, it calls errfunc with two arguments, like this: 

          (*errfunc) (filename, error-code)

The argument filename is the name of the directory that glob couldn't open or couldn't read, and error-code is the errno value that was reported to glob.

If the error handler function returns nonzero, then glob gives up right away. Otherwise, it continues.

GLOB_MARK
If the pattern matches the name of a directory, append `/' to the directory's name when returning it.
GLOB_NOCHECK
If the pattern doesn't match any file names, return the pattern itself as if it were a file name that had been matched. (Normally, when the pattern doesn't match anything, glob returns that there were no matches.)
GLOB_NOSORT
Don't sort the file names; return them in no particular order. (In practice, the order will depend on the order of the entries in the directory.) The only reason not to sort is to save time.
GLOB_NOESCAPE
Don't treat the `\' character specially in patterns. Normally, `\' quotes the following character, turning off its special meaning (if any) so that it matches only itself. When quoting is enabled, the pattern `\?' matches only the string `?', because the question mark in the pattern acts like an ordinary character.

If you use GLOB_NOESCAPE, then `\' is an ordinary character.

glob does its work by calling the function fnmatch repeatedly. It handles the flag GLOB_NOESCAPE by turning on the FNM_NOESCAPE flag in calls to fnmatch.

Previous: Flags for Globbing, Up: Globbing

10.2.3 More Flags for Globbing

Beside the flags described in the last section, the GNU implementation of glob allows a few more flags which are also defined in the glob.h file. Some of the extensions implement functionality which is available in modern shell implementations.

GLOB_PERIOD
The . character (period) is treated special. It cannot be matched by wildcards. See Wildcard Matching, FNM_PERIOD.
GLOB_MAGCHAR
The GLOB_MAGCHAR value is not to be given to glob in the flags parameter. Instead, glob sets this bit in the gl_flags element of the glob_t structure provided as the result if the pattern used for matching contains any wildcard character.
GLOB_ALTDIRFUNC
Instead of the using the using the normal functions for accessing the filesystem the glob implementation uses the user-supplied functions specified in the structure pointed to by pglob parameter. For more information about the functions refer to the sections about directory handling see Accessing Directories, and Reading Attributes.
GLOB_BRACE
If this flag is given the handling of braces in the pattern is changed. It is now required that braces appear correctly grouped. I.e., for each opening brace there must be a closing one. Braces can be used recursively. So it is possible to define one brace expression in another one. It is important to note that the range of each brace expression is completely contained in the outer brace expression (if there is one).

The string between the matching braces is separated into single expressions by splitting at , (comma) characters. The commas themselves are discarded. Please note what we said above about recursive brace expressions. The commas used to separate the subexpressions must be at the same level. Commas in brace subexpressions are not matched. They are used during expansion of the brace expression of the deeper level. The example below shows this 

          glob ("{foo/{,bar,biz},baz}", GLOB_BRACE, NULL, &result)

is equivalent to the sequence 

          glob ("foo/", GLOB_BRACE, NULL, &result)
          glob ("foo/bar", GLOB_BRACE|GLOB_APPEND, NULL, &result)
          glob ("foo/biz", GLOB_BRACE|GLOB_APPEND, NULL, &result)
          glob ("baz", GLOB_BRACE|GLOB_APPEND, NULL, &result)

if we leave aside error handling.

GLOB_NOMAGIC
If the pattern contains no wildcard constructs (it is a literal file name), return it as the sole “matching” word, even if no file exists by that name.
GLOB_TILDE
If this flag is used the character ~ (tilde) is handled special if it appears at the beginning of the pattern. Instead of being taken verbatim it is used to represent the home directory of a known user.

If ~ is the only character in pattern or it is followed by a / (slash), the home directory of the process owner is substituted. Using getlogin and getpwnam the information is read from the system databases. As an example take user bart with his home directory at /home/bart. For him a call like 

          glob ("~/bin/*", GLOB_TILDE, NULL, &result)

would return the contents of the directory /home/bart/bin. Instead of referring to the own home directory it is also possible to name the home directory of other users. To do so one has to append the user name after the tilde character. So the contents of user homer's bin directory can be retrieved by 

          glob ("~homer/bin/*", GLOB_TILDE, NULL, &result)

If the user name is not valid or the home directory cannot be determined for some reason the pattern is left untouched and itself used as the result. I.e., if in the last example home is not available the tilde expansion yields to "~homer/bin/*" and glob is not looking for a directory named ~homer.

This functionality is equivalent to what is available in C-shells if the nonomatch flag is set.

GLOB_TILDE_CHECK
If this flag is used glob behaves like as if GLOB_TILDE is given. The only difference is that if the user name is not available or the home directory cannot be determined for other reasons this leads to an error. glob will return GLOB_NOMATCH instead of using the pattern itself as the name.

This functionality is equivalent to what is available in C-shells if nonomatch flag is not set.

GLOB_ONLYDIR
If this flag is used the globbing function takes this as a hint that the caller is only interested in directories matching the pattern. If the information about the type of the file is easily available non-directories will be rejected but no extra work will be done to determine the information for each file. I.e., the caller must still be able to filter directories out.

This functionality is only available with the GNU glob implementation. It is mainly used internally to increase the performance but might be useful for a user as well and therefore is documented here.

Calling glob will in most cases allocate resources which are used to represent the result of the function call. If the same object of type glob_t is used in multiple call to glob the resources are freed or reused so that no leaks appear. But this does not include the time when all glob calls are done.

— Function: void globfree (glob_t *pglob)

The globfree function frees all resources allocated by previous calls to glob associated with the object pointed to by pglob. This function should be called whenever the currently used glob_t typed object isn't used anymore.

— Function: void globfree64 (glob64_t *pglob)

This function is equivalent to globfree but it frees records of type glob64_t which were allocated by glob64.

Next: , Previous: Globbing, Up: Pattern Matching

10.3 Regular Expression Matching

The GNU C library supports two interfaces for matching regular expressions. One is the standard POSIX.2 interface, and the other is what the GNU system has had for many years.

Both interfaces are declared in the header file regex.h. If you define _POSIX_C_SOURCE, then only the POSIX.2 functions, structures, and constants are declared.

Next: , Up: Regular Expressions

10.3.1 POSIX Regular Expression Compilation

Before you can actually match a regular expression, you must compile it. This is not true compilation—it produces a special data structure, not machine instructions. But it is like ordinary compilation in that its purpose is to enable you to “execute” the pattern fast. (See Matching POSIX Regexps, for how to use the compiled regular expression for matching.)

There is a special data type for compiled regular expressions:

— Data Type: regex_t

This type of object holds a compiled regular expression. It is actually a structure. It has just one field that your programs should look at:

re_nsub
This field holds the number of parenthetical subexpressions in the regular expression that was compiled.

There are several other fields, but we don't describe them here, because only the functions in the library should use them.

After you create a regex_t object, you can compile a regular expression into it by calling regcomp.

— Function: int regcomp (regex_t *restrict compiled, const char *restrict pattern, int cflags)

The function regcomp “compiles” a regular expression into a data structure that you can use with regexec to match against a string. The compiled regular expression format is designed for efficient matching. regcomp stores it into *compiled.

It's up to you to allocate an object of type regex_t and pass its address to regcomp.

The argument cflags lets you specify various options that control the syntax and semantics of regular expressions. See Flags for POSIX Regexps.

If you use the flag REG_NOSUB, then regcomp omits from the compiled regular expression the information necessary to record how subexpressions actually match. In this case, you might as well pass 0 for the matchptr and nmatch arguments when you call regexec.

If you don't use REG_NOSUB, then the compiled regular expression does have the capacity to record how subexpressions match. Also, regcomp tells you how many subexpressions pattern has, by storing the number in compiled->re_nsub. You can use that value to decide how long an array to allocate to hold information about subexpression matches.

regcomp returns 0 if it succeeds in compiling the regular expression; otherwise, it returns a nonzero error code (see the table below). You can use regerror to produce an error message string describing the reason for a nonzero value; see Regexp Cleanup.

Here are the possible nonzero values that regcomp can return:

REG_BADBR
There was an invalid `\{...\}' construct in the regular expression. A valid `\{...\}' construct must contain either a single number, or two numbers in increasing order separated by a comma.
REG_BADPAT
There was a syntax error in the regular expression.
REG_BADRPT
A repetition operator such as `?' or `*' appeared in a bad position (with no preceding subexpression to act on).
REG_ECOLLATE
The regular expression referred to an invalid collating element (one not defined in the current locale for string collation). See Locale Categories.
REG_ECTYPE
The regular expression referred to an invalid character class name.
REG_EESCAPE
The regular expression ended with `\'.
REG_ESUBREG
There was an invalid number in the `\digit' construct.
REG_EBRACK
There were unbalanced square brackets in the regular expression.
REG_EPAREN
An extended regular expression had unbalanced parentheses, or a basic regular expression had unbalanced `\(' and `\)'.
REG_EBRACE
The regular expression had unbalanced `\{' and `\}'.
REG_ERANGE
One of the endpoints in a range expression was invalid.
REG_ESPACE
regcomp ran out of memory.

Next: , Previous: POSIX Regexp Compilation, Up: Regular Expressions

10.3.2 Flags for POSIX Regular Expressions

These are the bit flags that you can use in the cflags operand when compiling a regular expression with regcomp.

REG_EXTENDED
Treat the pattern as an extended regular expression, rather than as a basic regular expression.
REG_ICASE
Ignore case when matching letters.
REG_NOSUB
Don't bother storing the contents of the matches-ptr array.
REG_NEWLINE
Treat a newline in string as dividing string into multiple lines, so that `$' can match before the newline and `^' can match after. Also, don't permit `.' to match a newline, and don't permit `[^...]' to match a newline.

Otherwise, newline acts like any other ordinary character.

Next: , Previous: Flags for POSIX Regexps, Up: Regular Expressions

10.3.3 Matching a Compiled POSIX Regular Expression

Once you have compiled a regular expression, as described in POSIX Regexp Compilation, you can match it against strings using regexec. A match anywhere inside the string counts as success, unless the regular expression contains anchor characters (`^' or `$').

— Function: int regexec (const regex_t *restrict compiled, const char *restrict string, size_t nmatch, regmatch_t matchptr[restrict], int eflags)

This function tries to match the compiled regular expression *compiled against string.

regexec returns 0 if the regular expression matches; otherwise, it returns a nonzero value. See the table below for what nonzero values mean. You can use regerror to produce an error message string describing the reason for a nonzero value; see Regexp Cleanup.

The argument eflags is a word of bit flags that enable various options.

If you want to get information about what part of string actually matched the regular expression or its subexpressions, use the arguments matchptr and nmatch. Otherwise, pass 0 for nmatch, and NULL for matchptr. See Regexp Subexpressions.

You must match the regular expression with the same set of current locales that were in effect when you compiled the regular expression.

The function regexec accepts the following flags in the eflags argument:

REG_NOTBOL
Do not regard the beginning of the specified string as the beginning of a line; more generally, don't make any assumptions about what text might precede it.
REG_NOTEOL
Do not regard the end of the specified string as the end of a line; more generally, don't make any assumptions about what text might follow it.

Here are the possible nonzero values that regexec can return:

REG_NOMATCH
The pattern didn't match the string. This isn't really an error.
REG_ESPACE
regexec ran out of memory.

Next: , Previous: Matching POSIX Regexps, Up: Regular Expressions

10.3.4 Match Results with Subexpressions

When regexec matches parenthetical subexpressions of pattern, it records which parts of string they match. It returns that information by storing the offsets into an array whose elements are structures of type regmatch_t. The first element of the array (index 0) records the part of the string that matched the entire regular expression. Each other element of the array records the beginning and end of the part that matched a single parenthetical subexpression.

— Data Type: regmatch_t

This is the data type of the matcharray array that you pass to regexec. It contains two structure fields, as follows:

rm_so
The offset in string of the beginning of a substring. Add this value to string to get the address of that part.
rm_eo
The offset in string of the end of the substring.

— Data Type: regoff_t

regoff_t is an alias for another signed integer type. The fields of regmatch_t have type regoff_t.

The regmatch_t elements correspond to subexpressions positionally; the first element (index 1) records where the first subexpression matched, the second element records the second subexpression, and so on. The order of the subexpressions is the order in which they begin.

When you call regexec, you specify how long the matchptr array is, with the nmatch argument. This tells regexec how many elements to store. If the actual regular expression has more than nmatch subexpressions, then you won't get offset information about the rest of them. But this doesn't alter whether the pattern matches a particular string or not.

If you don't want regexec to return any information about where the subexpressions matched, you can either supply 0 for nmatch, or use the flag REG_NOSUB when you compile the pattern with regcomp.

Next: , Previous: Regexp Subexpressions, Up: Regular Expressions

10.3.5 Complications in Subexpression Matching

Sometimes a subexpression matches a substring of no characters. This happens when `f\(o*\)' matches the string `fum'. (It really matches just the `f'.) In this case, both of the offsets identify the point in the string where the null substring was found. In this example, the offsets are both 1.

Sometimes the entire regular expression can match without using some of its subexpressions at all—for example, when `ba\(na\)*' matches the string `ba', the parenthetical subexpression is not used. When this happens, regexec stores -1 in both fields of the element for that subexpression.

Sometimes matching the entire regular expression can match a particular subexpression more than once—for example, when `ba\(na\)*' matches the string `bananana', the parenthetical subexpression matches three times. When this happens, regexec usually stores the offsets of the last part of the string that matched the subexpression. In the case of `bananana', these offsets are 6 and 8.

But the last match is not always the one that is chosen. It's more accurate to say that the last opportunity to match is the one that takes precedence. What this means is that when one subexpression appears within another, then the results reported for the inner subexpression reflect whatever happened on the last match of the outer subexpression. For an example, consider `\(ba\(na\)*s \)*' matching the string `bananas bas '. The last time the inner expression actually matches is near the end of the first word. But it is considered again in the second word, and fails to match there. regexec reports nonuse of the “na” subexpression.

Another place where this rule applies is when the regular expression 

     \(ba\(na\)*s \|nefer\(ti\)* \)*

matches `bananas nefertiti'. The “na” subexpression does match in the first word, but it doesn't match in the second word because the other alternative is used there. Once again, the second repetition of the outer subexpression overrides the first, and within that second repetition, the “na” subexpression is not used. So regexec reports nonuse of the “na” subexpression.

Previous: Subexpression Complications, Up: Regular Expressions

10.3.6 POSIX Regexp Matching Cleanup

When you are finished using a compiled regular expression, you can free the storage it uses by calling regfree.

— Function: void regfree (regex_t *compiled)

Calling regfree frees all the storage that *compiled points to. This includes various internal fields of the regex_t structure that aren't documented in this manual.

regfree does not free the object *compiled itself.

You should always free the space in a regex_t structure with regfree before using the structure to compile another regular expression.

When regcomp or regexec reports an error, you can use the function regerror to turn it into an error message string.

— Function: size_t regerror (int errcode, const regex_t *restrict compiled, char *restrict buffer, size_t length)

This function produces an error message string for the error code errcode, and stores the string in length bytes of memory starting at buffer. For the compiled argument, supply the same compiled regular expression structure that regcomp or regexec was working with when it got the error. Alternatively, you can supply NULL for compiled; you will still get a meaningful error message, but it might not be as detailed.

If the error message can't fit in length bytes (including a terminating null character), then regerror truncates it. The string that regerror stores is always null-terminated even if it has been truncated.

The return value of regerror is the minimum length needed to store the entire error message. If this is less than length, then the error message was not truncated, and you can use it. Otherwise, you should call regerror again with a larger buffer.

Here is a function which uses regerror, but always dynamically allocates a buffer for the error message: 

          char *get_regerror (int errcode, regex_t *compiled)
          {
            size_t length = regerror (errcode, compiled, NULL, 0);
            char *buffer = xmalloc (length);
            (void) regerror (errcode, compiled, buffer, length);
            return buffer;
          }

Previous: Regular Expressions, Up: Pattern Matching

10.4 Shell-Style Word Expansion

Word expansion means the process of splitting a string into words and substituting for variables, commands, and wildcards just as the shell does.

For example, when you write `ls -l foo.c', this string is split into three separate words—`ls', `-l' and `foo.c'. This is the most basic function of word expansion.

When you write `ls *.c', this can become many words, because the word `*.c' can be replaced with any number of file names. This is called wildcard expansion, and it is also a part of word expansion.

When you use `echo $PATH' to print your path, you are taking advantage of variable substitution, which is also part of word expansion.

Ordinary programs can perform word expansion just like the shell by calling the library function wordexp.

Next: , Up: Word Expansion

10.4.1 The Stages of Word Expansion

When word expansion is applied to a sequence of words, it performs the following transformations in the order shown here:

  1. Tilde expansion: Replacement of `~foo' with the name of the home directory of `foo'.
  2. Next, three different transformations are applied in the same step, from left to right:
  3. Field splitting: subdivision of the text into words.
  4. Wildcard expansion: The replacement of a construct such as `*.c' with a list of `.c' file names. Wildcard expansion applies to an entire word at a time, and replaces that word with 0 or more file names that are themselves words.
  5. Quote removal: The deletion of string-quotes, now that they have done their job by inhibiting the above transformations when appropriate.

For the details of these transformations, and how to write the constructs that use them, see The BASH Manual (to appear).

Next: , Previous: Expansion Stages, Up: Word Expansion

10.4.2 Calling wordexp

All the functions, constants and data types for word expansion are declared in the header file wordexp.h.

Word expansion produces a vector of words (strings). To return this vector, wordexp uses a special data type, wordexp_t, which is a structure. You pass wordexp the address of the structure, and it fills in the structure's fields to tell you about the results.

— Data Type: wordexp_t

This data type holds a pointer to a word vector. More precisely, it records both the address of the word vector and its size.

we_wordc
The number of elements in the vector.
we_wordv
The address of the vector. This field has type char **.
we_offs
The offset of the first real element of the vector, from its nominal address in the we_wordv field. Unlike the other fields, this is always an input to wordexp, rather than an output from it.

If you use a nonzero offset, then that many elements at the beginning of the vector are left empty. (The wordexp function fills them with null pointers.)

The we_offs field is meaningful only if you use the WRDE_DOOFFS flag. Otherwise, the offset is always zero regardless of what is in this field, and the first real element comes at the beginning of the vector.

— Function: int wordexp (const char *words, wordexp_t *word-vector-ptr, int flags)

Perform word expansion on the string words, putting the result in a newly allocated vector, and store the size and address of this vector into *word-vector-ptr. The argument flags is a combination of bit flags; see Flags for Wordexp, for details of the flags.

You shouldn't use any of the characters `|&;<>' in the string words unless they are quoted; likewise for newline. If you use these characters unquoted, you will get the WRDE_BADCHAR error code. Don't use parentheses or braces unless they are quoted or part of a word expansion construct. If you use quotation characters `'"`', they should come in pairs that balance.

The results of word expansion are a sequence of words. The function wordexp allocates a string for each resulting word, then allocates a vector of type char ** to store the addresses of these strings. The last element of the vector is a null pointer. This vector is called the word vector.

To return this vector, wordexp stores both its address and its length (number of elements, not counting the terminating null pointer) into *word-vector-ptr.

If wordexp succeeds, it returns 0. Otherwise, it returns one of these error codes:

WRDE_BADCHAR
The input string words contains an unquoted invalid character such as `|'.
WRDE_BADVAL
The input string refers to an undefined shell variable, and you used the flag WRDE_UNDEF to forbid such references.
WRDE_CMDSUB
The input string uses command substitution, and you used the flag WRDE_NOCMD to forbid command substitution.
WRDE_NOSPACE
It was impossible to allocate memory to hold the result. In this case, wordexp can store part of the results—as much as it could allocate room for.
WRDE_SYNTAX
There was a syntax error in the input string. For example, an unmatched quoting character is a syntax error.

— Function: void wordfree (wordexp_t *word-vector-ptr)

Free the storage used for the word-strings and vector that *word-vector-ptr points to. This does not free the structure *word-vector-ptr itself—only the other data it points to.

Next: , Previous: Calling Wordexp, Up: Word Expansion

10.4.3 Flags for Word Expansion

This section describes the flags that you can specify in the flags argument to wordexp. Choose the flags you want, and combine them with the C operator |.

WRDE_APPEND
Append the words from this expansion to the vector of words produced by previous calls to wordexp. This way you can effectively expand several words as if they were concatenated with spaces between them.

In order for appending to work, you must not modify the contents of the word vector structure between calls to wordexp. And, if you set WRDE_DOOFFS in the first call to wordexp, you must also set it when you append to the results.

WRDE_DOOFFS
Leave blank slots at the beginning of the vector of words. The we_offs field says how many slots to leave. The blank slots contain null pointers.
WRDE_NOCMD
Don't do command substitution; if the input requests command substitution, report an error.
WRDE_REUSE
Reuse a word vector made by a previous call to wordexp. Instead of allocating a new vector of words, this call to wordexp will use the vector that already exists (making it larger if necessary).

Note that the vector may move, so it is not safe to save an old pointer and use it again after calling wordexp. You must fetch we_pathv anew after each call.

WRDE_SHOWERR
Do show any error messages printed by commands run by command substitution. More precisely, allow these commands to inherit the standard error output stream of the current process. By default, wordexp gives these commands a standard error stream that discards all output.
WRDE_UNDEF
If the input refers to a shell variable that is not defined, report an error.

Next: , Previous: Flags for Wordexp, Up: Word Expansion

10.4.4 wordexp Example

Here is an example of using wordexp to expand several strings and use the results to run a shell command. It also shows the use of WRDE_APPEND to concatenate the expansions and of wordfree to free the space allocated by wordexp. 

     int
     expand_and_execute (const char *program, const char **options)
     {
       wordexp_t result;
       pid_t pid
       int status, i;
     
       /* Expand the string for the program to run.  */
       switch (wordexp (program, &result, 0))
         {
         case 0:			/* Successful.  */
           break;
         case WRDE_NOSPACE:
           /* If the error was WRDE_NOSPACE,
              then perhaps part of the result was allocated.  */
           wordfree (&result);
         default:                    /* Some other error.  */
           return -1;
         }
     
       /* Expand the strings specified for the arguments.  */
       for (i = 0; options[i] != NULL; i++)
         {
           if (wordexp (options[i], &result, WRDE_APPEND))
             {
               wordfree (&result);
               return -1;
             }
         }
     
       pid = fork ();
       if (pid == 0)
         {
           /* This is the child process.  Execute the command. */
           execv (result.we_wordv[0], result.we_wordv);
           exit (EXIT_FAILURE);
         }
       else if (pid < 0)
         /* The fork failed.  Report failure.  */
         status = -1;
       else
         /* This is the parent process.  Wait for the child to complete.  */
         if (waitpid (pid, &status, 0) != pid)
           status = -1;
     
       wordfree (&result);
       return status;
     }

Next: , Previous: Wordexp Example, Up: Word Expansion

10.4.5 Details of Tilde Expansion

It's a standard part of shell syntax that you can use `~' at the beginning of a file name to stand for your own home directory. You can use `~user' to stand for user's home directory.

Tilde expansion is the process of converting these abbreviations to the directory names that they stand for.

Tilde expansion applies to the `~' plus all following characters up to whitespace or a slash. It takes place only at the beginning of a word, and only if none of the characters to be transformed is quoted in any way.

Plain `~' uses the value of the environment variable HOME as the proper home directory name. `~' followed by a user name uses getpwname to look up that user in the user database, and uses whatever directory is recorded there. Thus, `~' followed by your own name can give different results from plain `~', if the value of HOME is not really your home directory.

Previous: Tilde Expansion, Up: Word Expansion

10.4.6 Details of Variable Substitution

Part of ordinary shell syntax is the use of `$variable' to substitute the value of a shell variable into a command. This is called variable substitution, and it is one part of doing word expansion.

There are two basic ways you can write a variable reference for substitution:

${variable}
If you write braces around the variable name, then it is completely unambiguous where the variable name ends. You can concatenate additional letters onto the end of the variable value by writing them immediately after the close brace. For example, `${foo}s' expands into `tractors'.
$variable
If you do not put braces around the variable name, then the variable name consists of all the alphanumeric characters and underscores that follow the `$'. The next punctuation character ends the variable name. Thus, `$foo-bar' refers to the variable foo and expands into `tractor-bar'.

When you use braces, you can also use various constructs to modify the value that is substituted, or test it in various ways.

${variable:-default}
Substitute the value of variable, but if that is empty or undefined, use default instead.
${variable:=default}
Substitute the value of variable, but if that is empty or undefined, use default instead and set the variable to default.
${variable:?message}
If variable is defined and not empty, substitute its value.

Otherwise, print message as an error message on the standard error stream, and consider word expansion a failure.

${variable:+replacement}
Substitute replacement, but only if variable is defined and nonempty. Otherwise, substitute nothing for this construct.
${#variable}
Substitute a numeral which expresses in base ten the number of characters in the value of variable. `${#foo}' stands for `7', because `tractor' is seven characters.

These variants of variable substitution let you remove part of the variable's value before substituting it. The prefix and suffix are not mere strings; they are wildcard patterns, just like the patterns that you use to match multiple file names. But in this context, they match against parts of the variable value rather than against file names.

${variable%%suffix}
Substitute the value of variable, but first discard from that variable any portion at the end that matches the pattern suffix.

If there is more than one alternative for how to match against suffix, this construct uses the longest possible match.

Thus, `${foo%%r*}' substitutes `t', because the largest match for `r*' at the end of `tractor' is `ractor'.

${variable%suffix}
Substitute the value of variable, but first discard from that variable any portion at the end that matches the pattern suffix.

If there is more than one alternative for how to match against suffix, this construct uses the shortest possible alternative.

Thus, `${foo%r*}' substitutes `tracto', because the shortest match for `r*' at the end of `tractor' is just `r'.

${variable##prefix}
Substitute the value of variable, but first discard from that variable any portion at the beginning that matches the pattern prefix.

If there is more than one alternative for how to match against prefix, this construct uses the longest possible match.

Thus, `${foo##*t}' substitutes `or', because the largest match for `*t' at the beginning of `tractor' is `tract'.

${variable#prefix}
Substitute the value of variable, but first discard from that variable any portion at the beginning that matches the pattern prefix.

If there is more than one alternative for how to match against prefix, this construct uses the shortest possible alternative.

Thus, `${foo#*t}' substitutes `ractor', because the shortest match for `*t' at the beginning of `tractor' is just `t'.

Next: , Previous: Pattern Matching, Up: Top

11 Input/Output Overview

Most programs need to do either input (reading data) or output (writing data), or most frequently both, in order to do anything useful. The GNU C library provides such a large selection of input and output functions that the hardest part is often deciding which function is most appropriate!

This chapter introduces concepts and terminology relating to input and output. Other chapters relating to the GNU I/O facilities are:

Next: , Up: I/O Overview

11.1 Input/Output Concepts

Before you can read or write the contents of a file, you must establish a connection or communications channel to the file. This process is called opening the file. You can open a file for reading, writing, or both. The connection to an open file is represented either as a stream or as a file descriptor. You pass this as an argument to the functions that do the actual read or write operations, to tell them which file to operate on. Certain functions expect streams, and others are designed to operate on file descriptors.

When you have finished reading to or writing from the file, you can terminate the connection by closing the file. Once you have closed a stream or file descriptor, you cannot do any more input or output operations on it.

Next: , Up: I/O Concepts

11.1.1 Streams and File Descriptors

When you want to do input or output to a file, you have a choice of two basic mechanisms for representing the connection between your program and the file: file descriptors and streams. File descriptors are represented as objects of type int, while streams are represented as FILE * objects.

File descriptors provide a primitive, low-level interface to input and output operations. Both file descriptors and streams can represent a connection to a device (such as a terminal), or a pipe or socket for communicating with another process, as well as a normal file. But, if you want to do control operations that are specific to a particular kind of device, you must use a file descriptor; there are no facilities to use streams in this way. You must also use file descriptors if your program needs to do input or output in special modes, such as nonblocking (or polled) input (see File Status Flags).

Streams provide a higher-level interface, layered on top of the primitive file descriptor facilities. The stream interface treats all kinds of files pretty much alike—the sole exception being the three styles of buffering that you can choose (see Stream Buffering).

The main advantage of using the stream interface is that the set of functions for performing actual input and output operations (as opposed to control operations) on streams is much richer and more powerful than the corresponding facilities for file descriptors. The file descriptor interface provides only simple functions for transferring blocks of characters, but the stream interface also provides powerful formatted input and output functions (printf and scanf) as well as functions for character- and line-oriented input and output.

Since streams are implemented in terms of file descriptors, you can extract the file descriptor from a stream and perform low-level operations directly on the file descriptor. You can also initially open a connection as a file descriptor and then make a stream associated with that file descriptor.

In general, you should stick with using streams rather than file descriptors, unless there is some specific operation you want to do that can only be done on a file descriptor. If you are a beginning programmer and aren't sure what functions to use, we suggest that you concentrate on the formatted input functions (see Formatted Input) and formatted output functions (see Formatted Output).

If you are concerned about portability of your programs to systems other than GNU, you should also be aware that file descriptors are not as portable as streams. You can expect any system running ISO C to support streams, but non-GNU systems may not support file descriptors at all, or may only implement a subset of the GNU functions that operate on file descriptors. Most of the file descriptor functions in the GNU library are included in the POSIX.1 standard, however.

Previous: Streams and File Descriptors, Up: I/O Concepts

11.1.2 File Position

One of the attributes of an open file is its file position that keeps track of where in the file the next character is to be read or written. In the GNU system, and all POSIX.1 systems, the file position is simply an integer representing the number of bytes from the beginning of the file.

The file position is normally set to the beginning of the file when it is opened, and each time a character is read or written, the file position is incremented. In other words, access to the file is normally sequential. Ordinary files permit read or write operations at any position within the file. Some other kinds of files may also permit this. Files which do permit this are sometimes referred to as random-access files. You can change the file position using the fseek function on a stream (see File Positioning) or the lseek function on a file descriptor (see I/O Primitives). If you try to change the file position on a file that doesn't support random access, you get the ESPIPE error. Streams and descriptors that are opened for append access are treated specially for output: output to such files is always appended sequentially to the end of the file, regardless of the file position. However, the file position is still used to control where in the file reading is done. If you think about it, you'll realize that several programs can read a given file at the same time. In order for each program to be able to read the file at its own pace, each program must have its own file pointer, which is not affected by anything the other programs do.

In fact, each opening of a file creates a separate file position. Thus, if you open a file twice even in the same program, you get two streams or descriptors with independent file positions.

By contrast, if you open a descriptor and then duplicate it to get another descriptor, these two descriptors share the same file position: changing the file position of one descriptor will affect the other.

Previous: I/O Concepts, Up: I/O Overview

11.2 File Names

In order to open a connection to a file, or to perform other operations such as deleting a file, you need some way to refer to the file. Nearly all files have names that are strings—even files which are actually devices such as tape drives or terminals. These strings are called file names. You specify the file name to say which file you want to open or operate on.

This section describes the conventions for file names and how the operating system works with them.

Next: , Up: File Names

11.2.1 Directories

In order to understand the syntax of file names, you need to understand how the file system is organized into a hierarchy of directories.

A directory is a file that contains information to associate other files with names; these associations are called links or directory entries. Sometimes, people speak of “files in a directory”, but in reality, a directory only contains pointers to files, not the files themselves.

The name of a file contained in a directory entry is called a file name component. In general, a file name consists of a sequence of one or more such components, separated by the slash character (`/'). A file name which is just one component names a file with respect to its directory. A file name with multiple components names a directory, and then a file in that directory, and so on.

Some other documents, such as the POSIX standard, use the term pathname for what we call a file name, and either filename or pathname component for what this manual calls a file name component. We don't use this terminology because a “path” is something completely different (a list of directories to search), and we think that “pathname” used for something else will confuse users. We always use “file name” and “file name component” (or sometimes just “component”, where the context is obvious) in GNU documentation. Some macros use the POSIX terminology in their names, such as PATH_MAX. These macros are defined by the POSIX standard, so we cannot change their names.

You can find more detailed information about operations on directories in File System Interface.

Next: , Previous: Directories, Up: File Names

11.2.2 File Name Resolution

A file name consists of file name components separated by slash (`/') characters. On the systems that the GNU C library supports, multiple successive `/' characters are equivalent to a single `/' character.

The process of determining what file a file name refers to is called file name resolution. This is performed by examining the components that make up a file name in left-to-right order, and locating each successive component in the directory named by the previous component. Of course, each of the files that are referenced as directories must actually exist, be directories instead of regular files, and have the appropriate permissions to be accessible by the process; otherwise the file name resolution fails.

If a file name begins with a `/', the first component in the file name is located in the root directory of the process (usually all processes on the system have the same root directory). Such a file name is called an absolute file name.

Otherwise, the first component in the file name is located in the current working directory (see Working Directory). This kind of file name is called a relative file name.

The file name components . (“dot”) and .. (“dot-dot”) have special meanings. Every directory has entries for these file name components. The file name component . refers to the directory itself, while the file name component .. refers to its parent directory (the directory that contains the link for the directory in question). As a special case, .. in the root directory refers to the root directory itself, since it has no parent; thus /.. is the same as /.

Here are some examples of file names:

/a
The file named a, in the root directory.
/a/b
The file named b, in the directory named a in the root directory.
a
The file named a, in the current working directory.
/a/./b
This is the same as /a/b.
./a
The file named a, in the current working directory.
../a
The file named a, in the parent directory of the current working directory.

A file name that names a directory may optionally end in a `/'. You can specify a file name of / to refer to the root directory, but the empty string is not a meaningful file name. If you want to refer to the current working directory, use a file name of . or ./.

Unlike some other operating systems, the GNU system doesn't have any built-in support for file types (or extensions) or file versions as part of its file name syntax. Many programs and utilities use conventions for file names—for example, files containing C source code usually have names suffixed with `.c'—but there is nothing in the file system itself that enforces this kind of convention.

Next: , Previous: File Name Resolution, Up: File Names

11.2.3 File Name Errors

Functions that accept file name arguments usually detect these errno error conditions relating to the file name syntax or trouble finding the named file. These errors are referred to throughout this manual as the usual file name errors.

EACCES
The process does not have search permission for a directory component of the file name.
ENAMETOOLONG
This error is used when either the total length of a file name is greater than PATH_MAX, or when an individual file name component has a length greater than NAME_MAX. See Limits for Files.

In the GNU system, there is no imposed limit on overall file name length, but some file systems may place limits on the length of a component.

ENOENT
This error is reported when a file referenced as a directory component in the file name doesn't exist, or when a component is a symbolic link whose target file does not exist. See Symbolic Links.
ENOTDIR
A file that is referenced as a directory component in the file name exists, but it isn't a directory.
ELOOP
Too many symbolic links were resolved while trying to look up the file name. The system has an arbitrary limit on the number of symbolic links that may be resolved in looking up a single file name, as a primitive way to detect loops. See Symbolic Links.

Previous: File Name Errors, Up: File Names