Source formatter

sultanc fmt gives SultanC source one canonical layout. The compiler command and the language server call the same library, compiler.format.format, so a file is formatted the same way wherever it is formatted. (std.fmt, the format strings of the standard library, is a different subject: Formatting.)

Contract#

• One canonical style. Indentation is four spaces; there is no option for the indent width or any other style setting.
• Line breaks are the author's. The formatter never wraps, joins or splits lines (module declarations, one line each, excepted); there is no line width.
• One formatter. sultanc fmt and the language server call the same library function and format identically. The language server applies no editor-specific rules and ignores editor indent settings.
• Meaning is kept. Tokens, comments, Arabic and Latin identifiers, keyword spellings in either language and every literal are kept byte for byte (details below).
• Deterministic and idempotent. The same input gives the same bytes on every run, and formatting formatted text changes nothing.
• Text in, text out. The formatter takes the text itself and reads nothing from disk; the path only names the source in diagnostics. An unsaved editor buffer formats like a file.
• All or nothing. A source that does not lex or parse gives its diagnostics and no output: never a partially formatted text, and a file is never partially rewritten.

What the formatter changes#

The formatter moves whitespace and nothing else.

• Every token is copied byte for byte and in order. Keywords stay in the language they were written in, deprecated spellings included, and identifiers, Arabic or Latin, are never respelled.
• Every comment is kept byte for byte, except for spaces at its end. Comments stay where they are: a comment on its own line stays on its own line, and a comment after code stays after that code.
• Text literals are never touched, including the lines inside a multi-line «…» literal.
• The direction marks LRM (U+200E), RLM (U+200F) and ALM (U+061C), in comments, in literals and between tokens, are kept byte for byte.

The author's line breaks are kept: apart from module declarations (below), the formatter never joins two lines or splits one, and there is no line width. What it decides is:

• Indentation. Four spaces per level. Statements inside {…} are one level inside the statement that opens the block. Lines inside (…) or […] are one level inside the line that opens the bracket. A line that starts with a closing bracket is at the level of its opening line. A line that continues a statement (the line before it did not end in ;, { or }, or in , inside a block) is one more level in, never more. The arms | pattern -> … of a match … with are at the level of the match, while the | lines of a type's variants are continuation lines. A line that starts with ; is at the level of its statement. An annotation line (@@test, @@register("/a")) is at the level of the declaration it annotates, and the declaration (or the next annotation) after it starts at the same column. An own-line comment is indented like the line after it.
• Spaces on a line. One space after ,, ; and :, around binary operators, =, := and ->, inside and around { }, between words, and between a keyword such as if or while and (. No space before ,, ;, :, ), ], . and ?, after (, [, ., @, ! and ~, between a callee and ( (f(x), uint(x)), before an index [, and after a prefix -, * or & (-1, *p, &mutable x).
• The author's choice, kept. <, > and >> either bracket type arguments (map<k, v>) or compare and shift, and the tokens alone do not say which, so the space the author wrote (none or one) stays. A < or > before a number, - or ! compares and is always spaced. A -, * or & written as a prefix right after a name (vector *u8) stays a prefix. The spaces before a trailing comment are kept, at least one, so aligned comments stay aligned.
• Blank lines. At most one blank line in a row; none at the start or the end of a file. The file ends with one line feed, CRLF line ends become LF, and no line ends in spaces.
• Module declarations. In the root file of a folder, each module / وحدة declaration is one line of its own: module io { ar: دخ, fr: entree_sortie };. The formatter joins a declaration written over several lines (a comment inside it keeps its line), starts it on a new line, and starts whatever follows its ; on the next line, a trailing comment excepted. This rule is the one place where the formatter moves a line break.

Formatting is idempotent: formatting a formatted file changes nothing.

Files that do not parse#

The formatter formats only a file the compiler can lex and parse. Any other file is reported with the diagnostics sultanc check prints for it, and nothing is written. This includes a file holding a raw bidirectional embedding, override or isolate control (U+202A to U+202E, U+2066 to U+2069) anywhere, in a comment or a literal too: the lexer rejects them, because they can make code display in an order other than the one compiled. A literal can still hold one written as an escape, and the formatter keeps the escape as written.

Semantics#

The formatter checks its own output every time, before anything is written: the output is lexed again by the compiler's lexer and must hold the same tokens (the same kinds and the same bytes, so the same values), the same comments and marks in the same order, and it must parse. If it does not, the formatter reports an internal compiler failure (SULTANC-ICE-COMPILER-FAILURE, stage formatter) and leaves the file as it was.

The parser reads tokens only, never the whitespace between them, so the same tokens give the same program. What can change is a position: a line number moves when blank lines are removed.

The command#

sultanc fmt main.sn lib/            # format in place; a directory means its .sn files
sultanc fmt --check main.sn lib/    # change nothing; report each file that is not formatted
sultanc fmt < main.sn               # standard input to standard output (also `sultanc fmt -`)

--check reports every file that is not formatted as SULTANC-FMT-UNFORMATTED, located at the first place the formatter would change. The exit status is 0 when every input is formatted (and, without --check, written); 1 when --check finds a file to format or an input does not lex or parse; 3 when an input cannot be found, read or written; and 2 for an invalid command line (--check without fmt, or a build option such as -o or --target with it). Standard input that does not parse writes nothing to standard output. A file is rewritten by writing a temporary file beside it and renaming that over it, so an interrupted run never leaves a partial file. --lang, --diagnostic-format and --color work as for every command.

Editor integration#

Editors and the language server call the library compiler.format.format directly. It takes the text itself: the path names the source in diagnostics only, and nothing is read from disk, so an unsaved buffer formats like a saved file. There is one style and no options; the formatter does not follow an editor's indent width, and there is no line width.

import compiler.format.format;

// The formatted text, or the diagnostics of a source that does not lex or parse.
def نسق_نص_مصدر(path: &string, text: &string): نتيجة_تنسيق
// The edits that format the text: UTF-8 byte offsets into it, in order and
// non-overlapping. No edit means the text is already formatted.
def تعديلات_تنسيق_نص(path: &string, text: &string): نتيجة_تعديلات_تنسيق
// The same edits, limited to those that touch the byte range [start, end].
def تعديلات_تنسيق_مدى(path: &string, text: &string, start: uint, end: uint): نتيجة_تعديلات_تنسيق

A call for a whole buffer:

import compiler.format.format;
import compiler.diagnostics.record;

def format_buffer(uri: &string, buffer: &string): option string {
    match نسق_نص_مصدر(uri, buffer) with
    | نتيجة_تنسيق.منسق(formatted) -> { return option.some(formatted); }
    | نتيجة_تنسيق.مرفوض(diagnostics) -> {
        // The source does not parse: `diagnostics` are the ones `check`
        // reports. There is no partial output; leave the buffer unchanged.
        return option.none();
    };
}

A result is either the whole formatted text (منسق) or the diagnostics (مرفوض), never partial text. An edit, تعديل_تنسيق, replaces the bytes [البداية, النهاية) of the input with النص. The formatter changes whitespace only, so every edit replaces the whitespace between two tokens or comments. Range formatting is exact: the indentation of a line depends on the whole file, and a range is formatted as the whole file formats it. A language server converts the byte offsets to its protocol's positions.

Verification#

The compiler sources and the standard library are the formatter's reference corpus. tools/tests/format_test.sh checks the command, and tools/tests/format_corpus.sh formats a copy of the whole corpus and checks that every file keeps its tokens, that a second run changes nothing, and that the compiler built from the formatted copy produces the same objects as the compiler built from the original.