Encoding (Internationalization best practices) - Revision history

TimRiker at 23:08, 23 February 2022

2022-02-23T23:08:05Z

← Older revision		Revision as of 23:08, 23 February 2022
Line 177:		Line 177:
	* Developer: Developers with less experience writing global software might want to validate a character by comparing it with character constants. Here are some Java examples:		* Developer: Developers with less experience writing global software might want to validate a character by comparing it with character constants. Here are some Java examples:

	<~~source~~ lang=java>		<syntaxhighlight lang=java>
	char ch;		char ch;
	//...		//...
Line 194:		Line 194:
	if ((ch == ' ') \|\| (ch =='\n') \|\| (ch == '\t'))		if ((ch == ' ') \|\| (ch =='\n') \|\| (ch == '\t'))
	// ...		// ...
	</~~source~~>		</syntaxhighlight>

	: This code (preceding) works ''only'' with English and a few other languages. To internationalize this example, replace it with the following statements:		: This code (preceding) works ''only'' with English and a few other languages. To internationalize this example, replace it with the following statements:

	<~~source~~ lang=java>		<syntaxhighlight lang=java>
	char ch;		char ch;
	// ...		// ...
Line 212:		Line 212:
	if (Character.isSpaceChar(ch))		if (Character.isSpaceChar(ch))
	// ...		// ...
	</~~source~~>		</syntaxhighlight>

	: The Character methods rely on the Unicode Standard determe character properties. In Java, char values represent Unicode characters. ''If you check char properties with the appropriate Character method, your code will work with all major languages.'' For example, the <code>Character.isLetter</code> method returns true if the character is a letter in Spanish, German, Chinese, Arabic, or another language.		: The Character methods rely on the Unicode Standard determe character properties. In Java, char values represent Unicode characters. ''If you check char properties with the appropriate Character method, your code will work with all major languages.'' For example, the <code>Character.isLetter</code> method returns true if the character is a letter in Spanish, German, Chinese, Arabic, or another language.

	: Here are some useful Character comparison methods. The Character API documentation fully specifies the methods.		: Here are some useful Character comparison methods. The Character API documentation fully specifies the methods.
	<~~source~~ lang=java>		<syntaxhighlight lang=java>
	isDigit		isDigit
	isLetter		isLetter
Line 225:		Line 225:
	isSpaceChar		isSpaceChar
	isDefined		isDefined
	</~~source~~>		</syntaxhighlight>

	: The <code>Character.getType</code> method returns the Unicode category of a character. Each category corresponds to a constant defined in the Character class. For example, for the character A, <code>getType</code> returns the <code>Character.UPPERCASE_LETTER</code> constant. For a complete list of the category constants, see the Character API documentation. Here is an example that uses the <code>getType</code> method and the Character category constants. All of the expressions in these if statements are true:		: The <code>Character.getType</code> method returns the Unicode category of a character. Each category corresponds to a constant defined in the Character class. For example, for the character A, <code>getType</code> returns the <code>Character.UPPERCASE_LETTER</code> constant. For a complete list of the category constants, see the Character API documentation. Here is an example that uses the <code>getType</code> method and the Character category constants. All of the expressions in these if statements are true:

	<~~source~~ lang=java>		<syntaxhighlight lang=java>
	if (Character.getType('a') == Character.LOWERCASE_LETTER)		if (Character.getType('a') == Character.LOWERCASE_LETTER)
	// ...		// ...
Line 241:		Line 241:
	if (Character.getType('_') == Character.CONNECTOR_PUNCTUATION)		if (Character.getType('_') == Character.CONNECTOR_PUNCTUATION)
	// ...		// ...
	</~~source~~>		</syntaxhighlight>

	* Tester: Ensure that character validation is being done properly.		* Tester: Ensure that character validation is being done properly.

TimRiker: utf8mb4 to fix wide unicode characters

2018-02-07T19:34:46Z

utf8mb4 to fix wide unicode characters

← Older revision		Revision as of 19:34, 7 February 2018
Line 30:		Line 30:
	** Oracle: Choose <code>AL32UTF8</code>		** Oracle: Choose <code>AL32UTF8</code>
	** SQL Server: Unicode compliance is managed in tables only		** SQL Server: Unicode compliance is managed in tables only
	** MySQL: default-character-set should be <code>~~utf8~~</code> and default-collation should be <code>~~utf8_unicode_ci~~</code>		** MySQL: default-character-set should be <code>utf8mb4</code> and default-collation should be <code>utf8mb4_unicode_ci</code>
	*** this should show no latin one output: <code>mysql -p -e 'show variables where `Variable_Name` LIKE "character_set%" OR `Variable_Name` LIKE "collation_%";'</code>		*** this should show no latin one output: <code>mysql -p -e 'show variables where `Variable_Name` LIKE "character_set%" OR `Variable_Name` LIKE "collation_%";'</code>
	*** sample settings for my.cnf		*** sample settings for my.cnf
Line 42:		Line 42:

	#TimRiker		#TimRiker
	default-character-set = ~~utf8~~		default-character-set = utf8mb4
	default-collation = ~~utf8_unicode_ci~~		default-collation = utf8mb4_unicode_ci
	skip-external-locking=1		skip-external-locking=1
	default-storage-engine = innodb		default-storage-engine = innodb
Line 53:		Line 53:

	[client]		[client]
	default-character-set=~~utf8~~		default-character-set=utf8mb4
	</pre>		</pre>

TimRiker: /* Unicode Compliance Checklist (draft) */

2013-12-18T18:11:22Z

Unicode Compliance Checklist (draft)

@@ Line 32: / Line 32: @@
 ** MySQL: default-character-set should be <code>utf8</code> and default-collation should be <code>utf8_unicode_ci</code>
 *** this should show no latin one output: <code>mysql -p -e 'show variables where `Variable_Name` LIKE "character_set%" OR `Variable_Name` LIKE "collation_%";'</code>
 * Table configuration: The database tables are Unicode compliant. For example:

TimRiker: /* Unicode Compliance Checklist (draft) */

2013-10-31T23:05:17Z

Unicode Compliance Checklist (draft)

← Older revision		Revision as of 23:05, 31 October 2013
Line 31:		Line 31:
	** SQL Server: Unicode compliance is managed in tables only		** SQL Server: Unicode compliance is managed in tables only
	** MySQL: default-character-set should be <code>utf8</code> and default-collation should be <code>utf8_unicode_ci</code>		** MySQL: default-character-set should be <code>utf8</code> and default-collation should be <code>utf8_unicode_ci</code>
	*** this should show no latin one output: <code>mysql -p -e 'show variables where `Variable_Name` LIKE "character_set%" OR `Variable_Name` LIKE "collation_%";'</code?		*** this should show no latin one output: <code>mysql -p -e 'show variables where `Variable_Name` LIKE "character_set%" OR `Variable_Name` LIKE "collation_%";'</code>

	* Table configuration: The database tables are Unicode compliant. For example:		* Table configuration: The database tables are Unicode compliant. For example:

TimRiker: /* Unicode Compliance Checklist (draft) */

2013-09-24T22:00:38Z

Unicode Compliance Checklist (draft)

← Older revision		Revision as of 22:00, 24 September 2013
Line 31:		Line 31:
	** SQL Server: Unicode compliance is managed in tables only		** SQL Server: Unicode compliance is managed in tables only
	** MySQL: default-character-set should be <code>utf8</code> and default-collation should be <code>utf8_unicode_ci</code>		** MySQL: default-character-set should be <code>utf8</code> and default-collation should be <code>utf8_unicode_ci</code>
			*** this should show no latin one output: <code>mysql -p -e 'show variables where `Variable_Name` LIKE "character_set%" OR `Variable_Name` LIKE "collation_%";'</code?

	* Table configuration: The database tables are Unicode compliant. For example:		* Table configuration: The database tables are Unicode compliant. For example:

Dspayne: /* Unicode Compliance Checklist (draft) */

2013-05-15T17:04:09Z

Unicode Compliance Checklist (draft)

← Older revision		Revision as of 17:04, 15 May 2013
Line 61:		Line 61:
	** Note: the above section seems like a good high-level objective that belongs in a general Internationalization document, and is not applicable to the decision to specifically use UTF-8.		** Note: the above section seems like a good high-level objective that belongs in a general Internationalization document, and is not applicable to the decision to specifically use UTF-8.
	* Only use Unicode string manipulation functions.		* Only use Unicode string manipulation functions.
	** Don't try to normalize your data sets at the string level. [~~Ask Stones what~~ this ~~means~~] Note: sometimes normalization can be helpful when searching across Unicode data, to normalize both the criteria and the return data so the system can recognize a match.		** Don't try to normalize your data sets at the string level. [clarify this] Note: sometimes normalization can be helpful when searching across Unicode data, to normalize both the criteria and the return data so the system can recognize a match.
	* Input: Allow at least any UTF-8 character to be input. (Verify composite/supplementary characters)		* Input: Allow at least any UTF-8 character to be input. (Verify composite/supplementary characters)
	* Output: Provide output in the encoding needed by the consuming software or hardware.		* Output: Provide output in the encoding needed by the consuming software or hardware.

Dspayne: /* Complex scripts */

2013-05-15T16:39:18Z

Complex scripts

← Older revision		Revision as of 16:39, 15 May 2013
Line 294:		Line 294:

	==Complex scripts ==		==Complex scripts ==

			----
	'''Best Practice:''' ''Handle complex scripts when necessary.''		'''Best Practice:''' ''Handle complex scripts when necessary.''

Dspayne: /* Space for bytes */

2013-05-15T16:39:09Z

Space for bytes

← Older revision		Revision as of 16:39, 15 May 2013
Line 280:		Line 280:

	==Space for bytes ==		==Space for bytes ==

			----
	'''Best Practice:''' ''Manage space to accommodate the number of bytes that result from coded character set conversions.''		'''Best Practice:''' ''Manage space to accommodate the number of bytes that result from coded character set conversions.''

Line 290:		Line 292:
	* Tester: <to be written>		* Tester: <to be written>
	'''Links:''' [http://www-01.ibm.com/software/globalization/guidelines/g5.html IBM]		'''Links:''' [http://www-01.ibm.com/software/globalization/guidelines/g5.html IBM]



	==Complex scripts ==		==Complex scripts ==

Dspayne: /* Multibyte character handling */

2013-05-15T16:39:00Z

Multibyte character handling

← Older revision		Revision as of 16:39, 15 May 2013
Line 266:		Line 266:

	==Multibyte character handling ==		==Multibyte character handling ==

			----
	'''Best Practice:''' ''Always treat a multibyte character as a unit.''		'''Best Practice:''' ''Always treat a multibyte character as a unit.''

Line 276:		Line 278:
	* Tester: <to be written>		* Tester: <to be written>
	'''Links:''' [http://www-01.ibm.com/software/globalization/guidelines/g3.html IBM]		'''Links:''' [http://www-01.ibm.com/software/globalization/guidelines/g3.html IBM]



	==Space for bytes ==		==Space for bytes ==

Dspayne: /* Multibyte character interpretation */

2013-05-15T16:38:48Z

Multibyte character interpretation

← Older revision		Revision as of 16:38, 15 May 2013
Line 252:		Line 252:

	==Multibyte character interpretation ==		==Multibyte character interpretation ==

			----
	'''Best Practice:''' ''Don't misinterpret a multibyte character as individual bytes.''		'''Best Practice:''' ''Don't misinterpret a multibyte character as individual bytes.''

Line 262:		Line 264:
	* Tester: <to be written>		* Tester: <to be written>
	'''Links:''' [http://www-01.ibm.com/software/globalization/guidelines/g2.html IBM]		'''Links:''' [http://www-01.ibm.com/software/globalization/guidelines/g2.html IBM]



	==Multibyte character handling ==		==Multibyte character handling ==