]>

mrankin@reactor.zone

CCSV media type tabular data This document specifies the Control-Character-Separated Values (CCSV) file format, a UTF-8 encoded tabular data format that uses ASCII control characters as field and record separators. CCSV is intended for exchanging structured, rectangular data between applications such as databases, spreadsheets, statistical software, and data conversion utilities. This document also specifies the text/csv media type for CCSV files. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Source for this draft and an issue tracker can be found at .

Introduction A CCSV (Control-Character-Separated Values) file is a file format for moving rectangular data between applications such as spreadsheets, statistical analysis programs, databases, and conversion utilities. CCSV is similar in purpose to Comma-Separated Values (CSV) files , Tab-Separated Values (TSV) files, and their derivatives. CCSV differs primarily in its use of non-printable ASCII control characters as field and record separators. Using control characters as separators reduces ambiguity when field values contain characters commonly used as delimiters in other formats. In particular, CCSV does not require quoting or escaping ordinary printable characters solely because those characters have special meaning in other delimited-value formats. A CCSV file consists of a required header followed by zero or more records. The header and every record contain the same number of fields. The first row therefore defines the names of the columns, while subsequent rows contain the corresponding values. This document defines the CCSV file format and registers the text/ccsv media type.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. The following terms are used throughout this document:

• Field: A single value withing a CCSV record.
• Record: A sequence of fields representing one row of data.
• Header: The first record in a CCSV file. The fields in the header identify the columns of the file.
• Unit Separator (US): The Unicode character U+001F, used to separate fields.
• Record Separator (RS): The Unicode character U+001E, used to separate records.

CCSV Format

File Structure A CCSV file MUST begin with a header. The header and each subsequent record MUST contain the same number of fields. A Unit Separator (US, U+001F) separates adjacent fields within a record. A Record Separator (RS, U+001E) separates adjacent records, including the header from the first data record. A CCSV file MUST therefore contain an RS following the header. An RS MAY also follow the final record. The presence or absence of an RS following the final record does not change the contents of the file. The end of the encoded CCSV data constitutes the end of the file. No additional end-of-file character or sequence is required.

Formatting Requirements A conforming CCSV file MUST satisfy all of the following requirements:

1. The file MUST be encoded in UTF-8 .
2. The file MUST NOT begin with a UTF-8 byte order mark (BOM, U+FEFF).
3. The file MUST begin with a header. The header consists of one or more fields separated by US (U+001F).
4. US (U+001F) MUST occur between adjacent fields in a record.
5. RS (U+001E) MUST occur between adjacent records, including between the header and the first data record.
6. An RS MAY occur immediately after the final record. A producer SHOULD NOT add an RS after the final record unless doing so is useful for the producer or its surrounding processing system.
7. The header and every record MUST contain the same number of fields.
8. Empty fields are represented by consecutive delimiters or by a delimiter adjacent to the beginning or end of a record, as applicable.
9. US (U+001F) and RS (U+001E) MUST NOT occur within the contents of a field.
10. Carriage return (CR, U+000D) and line feed (LF, U+000A) are valid field characters and MUST NOT be interpreted as record separators.

Except for US and RS, a field MAY contain any Unicode scalar value representable in UTF-8 including other ASCII control characters.

Syntax The following ABNF describes the CCSV structure at the UTF-8 octet level. The UTF-8 productions are taken from . The UTF-8 grammar above is provided for implementation convenience. The authoritative definition of UTF-8 is . A field MAY be empty. Consequently, the following are valid field arrangements when the surrounding record contains the appropriate number of fields. field2 field2 field1 ]]> where <US> represents U+001F.

Encoding Considerations CCSV files MUST be encoded using UTF-8 . The UTF-8 encoding applies to the entire CCSV file, including the header and all field contents. A CCSV file MUST NOT begin with a UTF-8 byte order mark (BOM, U+FEFF). UTF-8 permits CCSV to represent Unicode text while retaining the single-octet representation of the ASCII control characters used as delimiters. CR and LF are data characters in CCSV and are not record delimiters. Applications transporting CCSV content MUST preserve CR and LF occurring within fields as data. When CCSV is transported using a protocol or representation that imposes additional requirements on textual content, those requirements apply to the transport representation. Such processing MUST NOT alter the CCSV data represented by the payload.

Internationalization Considerations CCSV uses UTF-8 and therefore supports Unicode text in field and header values. CCSV does not define a language, locale, collation, numeric format, date format, or other interpretation of field contents. Applications processing CCSV data are responsible for applying appropriate locale-specific or application-specific interpretation to field values. Unicode strings are intended to be compared as identifiers or used in security-sensitive operations may require Unicode normalization or other application-specific processing. CCSV does not prescribe a normalization form.

Security Considerations CCSV is a passive data format and does not itself define executable content. However, CCSV data is commonly imported into applications such as spreadsheets, databases, statistical software, and scripts. Those applications may assign additional semantics to particular field values. In particular, applications that import CCSV data into spreadsheet software or other systems capable of evaluating expressions MUST NOT assume that field contents are trustworthy. An application may interpret a field beginning with a particular character as a formula, expression, command, or other executable content. Such applications SHOULD treat CCSV field contents as untrusted input and apply appropriate validation or escaping before passing the data to an interpreter. Implementations parsing CCSV data SHOULD also protect against resource-exhaustion attacks, including excessively large fields, records, or files. CCSV provides no confidentiality, integrity, authentication, or access-control mechanisms. Applications requiring those properties MUST provide them through mechanisms outside the CCSV format. Implementations MUST reject malformed UTF-8 rather than silently interpreting invalid UTF-8 sequences as valid CCSV data.

Interoperability Considerations Interoperability depends on consistent implementation of the CCSV formatting and encoding requirements. In particular, implementations MUST agree on all of the following:

• UTF-8 encoding;
• absence of a leading UTF-8 BOM;
• US (U+001F) as the field separator;
• RS (U+001E) as the record separator;
• equal field counts in the header and every record;
• representation of empty fields;
• prohibition of US and RS within field contents; and
• treatment of the CR and LF as field data rather than record separators.

Implementations MUST NOT treat printable delimiter characters as having special CCSV semantics. CCSV does not require quoting or escaping of printable characters merely because they have special meaning in other delimited-value formats. Implementations that consume CCSV SHOULD validate the file structure of the file before processing its contents.

IANA Considerations IANA is requested to register the following media type in the Standards Tree:

Type name:

text

Subtype name:

ccsv

Required parameters:

charset. The charset parameter MUST have the value "UTF-8".

Optional parameters:

None.

Encoding considerations:

8bit. CCSV content is encoded using UTF-8 as specified in . CR and LF occuring within fields are data characters and are not CCSV record separators.

Security considerations:

See .

Interoperability considerations:

See .

Published specification:

This document.

Applications that use this media type:

Databases, spreadsheets, statistical programs, data conversion utilities, and other applications that process rectangular tabular data.

Fragment identifier considerations:

CCSV does not define a fragment identifier syntax.

Additional information: :

Deprecated alias names for this type:

None.

Magic number(s):

None.

File name extension(s):

.ccsv

Macintosh file type code(s):

None.

Person & email address to contact for further information:

Mike rankin mrankin@oldgrognard.pub

Intended usage:

COMMON

Restrictions on usage:

None.

Author:

Mike Rankin mrankin@oldgrognard.pub

Change controller:

IETF

Examples In the examples below <US> represents U+001F and <RS> represents U+001E. The notation is used only to make the control characters visible and not part of an actual CCSV file

Basic Example The following CCSV data contains three columns and two records: AgeCity Alice42Boston Bob37Chicago ]]>

Empty Fields {empty-fields} The following record contains an empty second field: Boston ]]> The consecutive US characters represent an empty field. The following record begins with an empty field. 42Boston ]]>

Optional Final Record Separator Both of the following representations contain the same three fields and ar valid CCSV representations of a final record: 42Boston ]]> and: 42Boston ]]> A producer SHOULD generally omit the final RS unless its processing environment benefits from a terminating record separator.

Normative References This RFC documents the format used for Comma-Separated Values (CSV) files and registers the associated MIME type "text/csv". This memo provides information for the Internet community. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. ISO/IEC 10646-1 defines a large character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. The originally proposed encodings of the UCS, however, were not compatible with many current applications and protocols, and this has led to the development of UTF-8, the object of this memo. UTF-8 has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo obsoletes and replaces RFC 2279. Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK]

Acknowledgments The author thanks those individuals who provided feedback and technical discussions concerning the CCSV format and its proposed media type.