By default, WordPress uses URL with question mark and its post ID. After enable Pretty Permalinks, all the posts give 404 error. At first, I thought it’s .htaccess problem, however this file was updated automatically after applied the changes. After searching Wordpress documents, I found that it also requires FollowSymLinks option and AllowOverride FileInfo directives in Apache configuration file. I never knew this since shared hosting server has those options enabled by default. On VPS server, I have to configure by myself.
# Each directory to which Apache has access can be configured with respect
# to which services and features are allowed and/or disabled in that
# directory (and its subdirectories).
Allow from all