Chapter 3. Lexical Structure词法结构

Programs are written using the Unicode character set (1.7). 程序使用Unicode字符集（1.7）编写。Information about this character set and its associated character encodings may be found at https://www.unicode.org/.有关此字符集及其关联字符编码的信息，请访问https://www.unicode.org/。

The Java SE Platform tracks the Unicode Standard as it evolves. JavaSE平台跟踪Unicode标准的发展。The precise version of Unicode used by a given release is specified in the documentation of the class Character.类Character的文档中指定了给定版本使用的Unicode的精确版本。

Versions of the Java programming language prior to JDK 1.1 used Unicode 1.1.5. JDK1.1之前的Java编程语言版本使用Unicode 1.1.5。Upgrades to newer versions of the Unicode Standard occurred in JDK 1.1 (to Unicode 2.0), JDK 1.1.7 (to Unicode 2.1), Java SE 1.4 (to Unicode 3.0), Java SE 5.0 (to Unicode 4.0), Java SE 7 (to Unicode 6.0), Java SE 8 (to Unicode 6.2), Java SE 9 (to Unicode 8.0), Java SE 11 (to Unicode 10.0), Java SE 12 (to Unicode 11.0), Java SE 13 (to Unicode 12.1), and Java SE 15 (to Unicode 13.0).在JDK1.1（到Unicode 2.0）、JDK1.1.7（到Unicode 2.1）、JavaSE1.4（到Unicode 3.0）、JavaSE5.0（到Unicode 4.0）、JavaSE7（到Unicode 6.0）、JavaSE8（到Unicode 6.2）、JavaSE9（到Unicode 8.0）、JavaSE11（到Unicode 10.0）、JavaSE12（到Unicode 11.0）、JavaSE13（到Unicode 12.1）中升级到Unicode标准的更新版本，和JavaSE15（到Unicode 13.0）。

The Unicode standard was originally designed as a fixed-width 16-bit character encoding. Unicode标准最初设计为固定宽度的16位字符编码。It has since been changed to allow for characters whose representation requires more than 16 bits. The range of legal code points is now U+0000 to U+10FFFF, using the hexadecimal U+n notation. 此后，它被修改为允许表示需要超过16位的字符。使用十六进制U+n表示法，法定代码点的范围现在是U+0000到U+10FFFF。Characters whose code points are greater than U+FFFF are called supplementary characters. 代码点大于U+FFFF的字符称为补充字符。To represent the complete range of characters using only 16-bit units, the Unicode standard defines an encoding called UTF-16. 为了仅使用16位单位表示整个字符范围，Unicode标准定义了一种称为UTF-16的编码。In this encoding, supplementary characters are represented as pairs of 16-bit code units, the first from the high-surrogates range (U+D800 to U+DBFF), and the second from the low-surrogates range (U+DC00 to U+DFFF). 在这种编码中，补充字符表示为16位代码单元对，第一个来自高代理项范围（U+D800到U+DBFF），第二个来自低代理项范围（U+DC00到U+DFFF）。For characters in the range U+0000 to U+FFFF, the values of code points and UTF-16 code units are the same.对于U+0000到U+FFFF范围内的字符，代码点和UTF-16代码单位的值相同。

The Java programming language represents text in sequences of 16-bit code units, using the UTF-16 encoding.Java编程语言使用UTF-16编码以16位代码单元的序列表示文本。

Some APIs of the Java SE Platform, primarily in the Character class, use 32-bit integers to represent code points as individual entities. JavaSE平台的一些API（主要是Character类）使用32位整数将代码点表示为单个实体。The Java SE Platform provides methods to convert between 16-bit and 32-bit representations.JavaSE平台提供了在16位和32位表示之间转换的方法。

This specification uses the terms code point and UTF-16 code unit where the representation is relevant, and the generic term character where the representation is irrelevant to the discussion.本规范使用术语代码点和UTF-16代码单元（表示相关），以及通用术语字符（表示与讨论无关）。

Except for comments (3.7), identifiers (3.8), and the contents of character literals, string literals, and text blocks (3.10.4, 3.10.5, 3.10.6), all input elements (3.5) in a program are formed only from ASCII characters (or Unicode escapes (3.3) which result in ASCII characters).除了注释（3.7）、标识符（3.8）以及字符文本、字符串文本和文本块（3.10.4、3.10.5、3.10.6）的内容外，程序中的所有输入元素（3.5）仅由ASCII字符（或导致ASCII字符的Unicode转义（3.3））构成。

ASCII (ANSI X3.4) is the American Standard Code for Information Interchange. ASCII(ANSI X3.4）是信息交换的美国标准代码。The first 128 characters of the Unicode UTF-16 encoding are the ASCII characters.Unicode UTF-16编码的前128个字符是ASCII字符。

A raw Unicode character stream is translated into a sequence of tokens, using the following three lexical translation steps, which are applied in turn:原始Unicode字符流使用以下三个词汇转换步骤转换为一系列标记，这三个步骤依次应用：

The longest possible translation is used at each step, even if the result does not ultimately make a correct program while another lexical translation would. 每一步都使用尽可能长的翻译，即使结果最终不能生成正确的程序，而另一个词汇翻译可能会生成正确的程序。There are two exceptions to account for situations that need more granular translation: in step 1, for the processing of contiguous \ characters (3.3), and in step 3, for the processing of contextual keywords and adjacent > characters (3.5).对于需要更精细翻译的情况，有两种例外情况：在步骤1中，用于处理连续\字符（3.3），在步骤3中，用于处理上下文关键字和相邻>字符（3.5）。

The input characters a--b are tokenized as a, --, and b, which is not part of any grammatically correct program, even though the tokenization a, -, -, b could be part of a grammatically correct program. 输入字符a--b被标记为a，--和b，这不是任何语法正确程序的一部分，即使标记化a、-、-、b可能是语法正确程序的一部分。The tokenization a, -, -, b can be realized with the input characters a- -b (with an ASCII SP character between the two - characters).可以使用输入字符a- -b（两个-字符之间有一个ASCII SP字符）实现标记化a、-、-、-、b。

It might be supposed that the raw input \\u1234 is translated to a \ character and (following the "longest possible" rule) a Unicode escape of the form \u1234. 可以假定原始输入\\u1234被转换为\字符和（遵循“尽可能长的”规则）形式为\u1234的Unicode转义。In fact, the leading \ character causes this raw input to be translated to seven distinct characters:事实上，前导\字符导致此原始输入被转换为七个不同的字符： \ \ u 1 2 3 4.

A compiler for the Java programming language ("Java compiler") first recognizes Unicode escapes in its raw input, translating the ASCII characters \u followed by four hexadecimal digits to a raw input character which denotes the UTF-16 code unit (3.1) for the indicated hexadecimal value. Java编程语言的编译器（“Java编译器”）首先识别其原始输入中的Unicode转义，将ASCII字符\u后跟四个十六进制数字转换为原始输入字符，表示所示十六进制值的UTF-16代码单元（3.1）。One Unicode escape can represent characters in the range U+0000 to U+FFFF; representing supplementary characters in the range U+010000 to U+10FFFF requires two consecutive Unicode escapes. 一个Unicode转义可以表示U+0000到U+FFFF范围内的字符；表示U+010000到U+10FFFF范围内的补充字符需要两个连续的Unicode转义。All other characters in the compiler's raw input are recognized as raw input characters and passed unchanged.编译器原始输入中的所有其他字符都会被识别为原始输入字符，并以不变的方式传递。

This translation step results in a sequence of Unicode input characters, all of which are raw input characters (any Unicode escapes having been reduced to raw input characters).此转换步骤产生一系列Unicode输入字符，所有这些字符都是原始输入字符（任何Unicode转义都已减少为原始输入字符）。

The \, u, and hexadecimal digits here are all ASCII characters.这里的\、u和十六进制数字都是ASCII字符。

The UnicodeInputCharacter production is ambiguous because an ASCII \ character in the compiler's raw input could be reduced to either a RawInputCharacter or the \ of a UnicodeEscape (to be followed by an ASCII u). UnicodeInputCharacter产品是不明确的，因为编译器原始输入中的ASCII \字符可以减少为RawInputCharacter或UnicodeEscape的\字符（后面跟一个ASCII u）。To avoid ambiguity, for each ASCII \ character in the compiler's raw input, input processing must consider the most recent raw input characters that resulted from this translation step:为了避免歧义，对于编译器的原始输入中的每个ASCII字符\，输入处理必须考虑由该翻译步骤产生的最新的原始输入字符：

If an eligible \ is not followed by u, then it is treated as a RawInputCharacter and remains part of the escaped Unicode stream.如果合格的\后面没有u，则它将被视为RawInputCharacter，并且仍然是转义Unicode流的一部分。

If an eligible \ is followed by u, or more than one u, and the last u is not followed by four hexadecimal digits, then a compile-time error occurs.如果一个合格的\后面跟有u，或多个u，而最后一个u后面没有四个十六进制数字，则会发生编译时错误。

The character produced by a Unicode escape does not participate in further Unicode escapes.Unicode转义生成的字符不参与进一步的Unicode转义。

The Java programming language specifies a standard way of transforming a program written in Unicode into ASCII that changes a program into a form that can be processed by ASCII-based tools. Java编程语言指定了一种将Unicode编写的程序转换为ASCII的标准方法，该方法将程序转换为可由基于ASCII的工具处理的形式。The transformation involves converting any Unicode escapes in the source text of the program to ASCII by adding an extra u - for example, \uxxxx becomes \uuxxxx - while simultaneously converting non-ASCII characters in the source text to Unicode escapes containing a single u each.转换涉及将程序源文本中的任何Unicode转义转换为ASCII，方法是添加一个额外的u（例如，\uxxxx变为\uuxxxx），同时将源文本中的非ASCII字符转换为包含单个u的Unicode转义。

This transformed version is equally acceptable to a Java compiler and represents the exact same program. 这个转换后的版本同样可以被Java编译器接受，并且表示完全相同的程序。The exact Unicode source can later be restored from this ASCII form by converting each escape sequence where multiple u's are present to a sequence of Unicode characters with one fewer u, while simultaneously converting each escape sequence with a single u to the corresponding single Unicode character.通过将存在多个u的每个转义序列转换为少一个u的Unicode字符序列，同时将每个单u转义序列转换为相应的单Unicode字符，可以从这种ASCII形式恢复确切的Unicode源。

A Java compiler should use the \uxxxx notation as an output format to display Unicode characters when a suitable font is not available.当合适的字体不可用时，Java编译器应使用\uxxxx符号作为输出格式来显示Unicode字符。

A Java compiler next divides the sequence of Unicode input characters into lines by recognizing line terminators.Java编译器接下来通过识别行终止符将Unicode输入字符序列分成行。

Lines are terminated by the ASCII characters CR, or LF, or CR LF. 行由ASCII字符CR、LF或CR LF终止。The two characters CR immediately followed by LF are counted as one line terminator, not two.紧跟LF的两个字符CR被算作一行结束符，而不是两行结束符。

A line terminator specifies the termination of the // form of a comment (3.7).行终止符指定注释//形式的终止（3.7）。

The lines defined by line terminators may determine the line numbers produced by a Java compiler.由行终止符定义的行可以确定Java编译器生成的行号。

The result is a sequence of line terminators and input characters, which are the terminal symbols for the third step in the tokenization process.结果是一系列行终止符和输入字符，它们是标记化过程中第三步的终端符号。

The input characters and line terminators that result from Unicode escape processing (3.3) and then input line recognition (3.4) are reduced to a sequence of input elements.Unicode转义处理（3.3）和输入行识别（3.4）产生的输入字符和行终止符被简化为一个输入元素序列。

Those input elements that are not white space or comments are tokens. 那些不是空白或注释的输入元素是口令。The tokens are the terminal symbols of the syntactic grammar (2.3).标记是句法语法的终端符号（2.3）。

White space (3.6) and comments (3.7) can serve to separate tokens that, if adjacent, might be tokenized in another manner.空白（3.6）和注释（3.7）可以用来分隔标记，如果相邻，可能会以另一种方式标记。

For example, the input characters - and = can form the operator token -= (3.12) only if there is no intervening white space or comment. As another example, the ten input characters staticvoid form a single identifier token while the eleven input characters static void (with an ASCII SP character between c and v) form a pair of keyword tokens, static and void, separated by white space.

As a special concession for compatibility with certain operating systems, the ASCII SUB character (\u001a, or control-Z) is ignored if it is the last character in the escaped input stream.作为与某些操作系统兼容的特殊让步，如果ASCII子字符（\u001a，或control-Z）是转义输入流中的最后一个字符，则忽略它。

The Input production is ambiguous, meaning that for some sequences of input characters, there is more than one way to reduce the input characters to input elements (that is, to tokenize the input characters). 输入结果是不明确的，这意味着对于某些输入字符序列，有不止一种方法可以将输入字符简化为输入元素（即，将输入字符标记化）。Ambiguities are resolved as follows:歧义的解决方法如下：

Consider two tokens x and y in the resulting input stream. If x precedes y, then we say that x is to the left of y and that y is to the right of x.

class Empty {
}

White space is defined as the ASCII space character, horizontal tab character, form feed character, and line terminator characters (3.4).空白被定义为ASCII空格字符、水平制表符、换行符和行结束符（3.4）。

There are two kinds of comments:有两种评论：

These productions imply all of the following properties:这些产品包含以下所有属性：

/* this comment /* // /** ends here: */

The lexical grammar implies that comments do not occur within character literals, string literals, or text blocks (3.10.4, 3.10.5, 3.10.6).词汇语法意味着注释不会出现在字符、字符串或文本块中（3.10.4、3.10.5、3.10.6）。

An identifier is an unlimited-length sequence of Java letters and Java digits, the first of which must be a Java letter.

A "Java letter" is a character for which the method Character.isJavaIdentifierStart(int) returns true.

A "Java letter-or-digit" is a character for which the method Character.isJavaIdentifierPart(int) returns true.

The "Java letters" include uppercase and lowercase ASCII Latin letters A-Z (\u0041-\u005a), and a-z (\u0061-\u007a), and, for historical reasons, the ASCII dollar sign ($, or \u0024) and underscore (_, or \u005f). The dollar sign should be used only in mechanically generated source code or, rarely, to access pre-existing names on legacy systems. The underscore may be used in identifiers formed of two or more characters, but it cannot be used as a one-character identifier due to being a keyword.

The "Java digits" include the ASCII digits 0-9 (\u0030-\u0039).

Letters and digits may be drawn from the entire Unicode character set, which supports most writing scripts in use in the world today, including the large sets for Chinese, Japanese, and Korean. This allows programmers to use identifiers in their programs that are written in their native languages.

Two identifiers are the same only if, after ignoring characters that are ignorable, the identifiers have the same Unicode character for each letter or digit. An ignorable character is a character for which the method Character.isIdentifierIgnorable(int) returns true. Identifiers that have the same external appearance may yet be different.

An identifier never has the same spelling (Unicode character sequence) as a reserved keyword (3.9), a boolean literal (3.10.3) or the null literal (3.10.8), due to the rules of tokenization (3.5). However, an identifier may have the same spelling as a contextual keyword, because the tokenization of a sequence of input characters as an identifier or a contextual keyword depends on where the sequence appears in the program.

To facilitate the recognition of contextual keywords, the syntactic grammar (2.3) sometimes disallows certain identifiers by defining a production to accept only a subset of identifiers. The subsets are as follows:

TypeIdentifier is used in the declaration of classes, interfaces, and type parameters (8.1, 9.1, 4.4), and when referring to types (6.5). For example, the name of a class must be a TypeIdentifier, so it is illegal to declare a class named permits, record, sealed, var, or yield.

UnqualifiedMethodIdentifier is used when a method invocation expression refers to a method by its simple name (6.5.7.1). Since the term yield is excluded from UnqualifiedMethodIdentifier, any invocation of a method named yield must be qualified, thus distinguishing the invocation from a yield statement (14.21).

51 character sequences, formed from ASCII characters, are reserved for use as keywords and cannot be used as identifiers (3.8). Another 16 character sequences, also formed from ASCII characters, may be interpreted as keywords or as other tokens, depending on the context in which they appear.

abstract   continue   for      new     switch
assert   default  if       package   synchronized
boolean  do     goto     private   this
break    double   implements   protected   throw
byte     else     import     public    throws
case     enum     instanceof   return    transient
catch    extends  int      short     try
char     final    interface  static    void
class    finally  long     strictfp  volatile
const    float    native     super     while
_ (underscore)

exports    opens    requires   uses
module     permits  sealed     var
non-sealed   provides   to       with
open     record   transitive   yield

The keywords const and goto are reserved, even though they are not currently used. This may allow a Java compiler to produce better error messages if these C++ keywords incorrectly appear in programs.

The keyword strictfp is obsolete and should not be used in new code.

The keyword _ (underscore) is reserved for possible future use in parameter declarations.

true and false are not keywords, but rather boolean literals (3.10.3).

null is not a keyword, but rather the null literal (3.10.8).

During the reduction of input characters to input elements (3.5), a sequence of input characters that notionally matches a contextual keyword is reduced to a contextual keyword if and only if both of the following conditions hold:

In general, accidentally omitting white space in source code will cause a sequence of input characters to be tokenized as an identifier, due to the "longest possible translation" rule (3.2). For example, the sequence of twelve input characters p u b l i c s t a t i c is always tokenized as the identifier publicstatic, rather than as the reserved keywords public and static. If two tokens are intended, they must be separated by white space or a comment.

The rule above works in tandem with the "longest possible translation" rule to produce an intuitive result in contexts where contextual keywords may appear. For example, the sequence of eleven input characters v a r f i l e n a m e is usually tokenized as the identifier varfilename, but in a local variable declaration, the first three input characters are tentatively recognized as the contextual keyword var by the first condition of the rule above. However, it would be confusing to overlook the lack of white space in the sequence by recognizing the next eight input characters as the identifier filename. (This would mean that the sequence undergoes different tokenization in different contexts: an identifier in most contexts, but a contextual keyword and an identifier in local variable declarations.) Accordingly, the second condition prevents recognition of the contextual keyword var on the grounds that the immediately following input character f is a JavaLetterOrDigit. The sequence v a r f i l e n a m e is therefore tokenized as the identifier varfilename in a local variable declaration.

As another example of the careful recognition of contextual keywords, consider the sequence of 15 input characters n o n - s e a l e d c l a s s. This sequence is usually translated to three tokens - the identifier non, the operator -, and the identifier sealedclass - but in a normal class declaration, where the first condition holds, the first ten input characters are tentatively recognized as the contextual keyword non-sealed. To avoid translating the sequence to two keyword tokens (non-sealed and class) rather than three non-keyword tokens, and to avoid rewarding the programmer for omitting white space before class, the second condition prevents recognition of the contextual keyword. The sequence n o n - s e a l e d c l a s s is therefore tokenized as three tokens in a class declaration.

In the rule above, the first condition depends on details of the syntactic grammar, but a compiler for the Java programming language can implement the rule without fully parsing the input program. For example, a heuristic could be used to track the contextual state of the tokenizer, as long as the heuristic guarantees that valid uses of contextual keywords are tokenized as keywords, and valid uses of identifiers are tokenized as identifiers. Alternatively, a compiler could always tokenize a contextual keyword as an identifier, leaving it to a later phase to recognize special uses of these identifiers.

A literal is the source code representation of a value of a primitive type (4.2), the String type (4.3.3), or the null type (4.1).

0  2  0372  0xDada_Cafe  1996  0x00_FF__00_FF

0l  0777L  0x100000000L  2_147_483_648L  0xC0B0L

1e1f  2.f  .3f  0f  3.14f  6.022137e+23f

1e1  2.  .3  0.0  3.14  1e-9d  1e137

""          // the empty string
"\""          // a string containing " alone
"This is a string"  // a string containing 16 characters
"This is a " +    // actually a string-valued constant expression,
  "two-line string"  // formed from two string literals

package testPackage;
class Test {
  public static void main(String[] args) {
    String hello = "Hello", lo = "lo";
    System.out.println(hello == "Hello");
    System.out.println(Other.hello == hello);
    System.out.println(other.Other.hello == hello);
    System.out.println(hello == ("Hel"+"lo"));
    System.out.println(hello == ("Hel"+lo));
    System.out.println(hello == ("Hel"+lo).intern());
  }
}
class Other { static String hello = "Hello"; }

package other;
public class Other { public static String hello = "Hello"; }

true
true
true
true
false
true

String html = "<html>\n" +
        "  <body>\n" +
        "    <p>Hello, world</p>\n" +
        "  </body>\n" +
        "</html>\n";

String html = """
        <html>
          <body>
            <p>Hello, world</p>
          </body>
        </html>
        """;

class Test {
  public static void main(String[] args) {
    // The six characters w i n t e r
    String season = """
            winter""";

    // The seven characters w i n t e r LF
    String period = """
            winter
            """;

    // The ten characters H i , SP " B o b " LF
    String greeting = """
              Hi, "Bob"
              """;

    // The eleven characters H i , LF SP " B o b " LF
    String salutation = """
              Hi,
               "Bob"
              """;    

    // The empty string (zero length)
    String empty = """
             """;    

    // The two characters " LF
    String quote = """
             "
             """; 

    // The two characters \ LF
    String backslash = """
               \\
               """;  
  }
}

class Story1 {
  public static void main(String[] args) {
    String story = """
      "When I use a word," Humpty Dumpty said,
      in rather a scornful tone, "it means just what I
      choose it to mean - neither more nor less."
      "The question is," said Alice, "whether you
      can make words mean so many different things."
      "The question is," said Humpty Dumpty,
      "which is to be master - that's all."
    """;
  }
}

class Story2 {
  public static void main(String[] args) {
    String story = """
      "When I use a word," Humpty Dumpty said,
      in rather a scornful tone, "it means just what I
      choose it to mean - neither more nor less."
      "The question is," said Alice, "whether you
      can make words mean so many different things."
      "The question is," said Humpty Dumpty,
      "which is to be master - that's all."""";  // error
  }
}

class Story3 {
  public static void main(String[] args) {
    String story = """
      "When I use a word," Humpty Dumpty said,
      in rather a scornful tone, "it means just what I
      choose it to mean - neither more nor less."
      "The question is," said Alice, "whether you
      can make words mean so many different things."
      "The question is," said Humpty Dumpty,
      "which is to be master - that's all.\"""";  // OK
  }
}

class Code {
  public static void main(String[] args) {
    String text = """
      The quick brown fox jumps over the lazy dog
    """;

    String code =
      """
      String text = \"""
        The quick brown fox jumps over the lazy dog
      \""";
      """;
  }
}

String html = """
        <html>\r
          <body>\r
            <p>Hello, world</p>\r
          </body>\r
        </html>\r
        """;

|<html>\u000D\u000A
|  <body>\u000D\u000A
|    <p>Hello, world</p>\u000D\u000A
|  </body>\u000D\u000A
|</html>\u000D\u000A

System.out.println("ab" + """
              cde
              """);

String cde = """
       abcde""".substring(2);

String math = """
        1+1 equals \
        """ + String.valueOf(2);

@Preconditions("""
  rate > 0 &&
  rate <= MAX_REFRESH_RATE
""")
public void setRefreshRate(int rate) { ... }

Twelve tokens, formed from ASCII characters, are the separators (punctuators).

(   )   {   }   [   ]   ;   ,   .   ...   @   ::

38 tokens, formed from ASCII characters, are the operators.

=   >   <   !   ~   ?   :   ->
==  >=  <=  !=  &&  ||  ++  --
+   -   *   /   &   |   ^   %   <<   >>   >>>
+=  -=  *=  /=  &=  |=  ^=  %=  <<=  >>=  >>>=