Drupal claims that updating sites from D8 to D9 is easy. That is true if you have no additional modules installed. But in the real world, it is really difficult. I have done 4 sites now, and all were somewhat different, alas. The issue is updating the modules, or removing those that cannot be updated to D9. Here is my procedure.

Make sure your D8 site is fully updated

Update everything to the latest Drupal8 (currently 8.9.6) version.

Install the upgrade_status module

https://www.drupal.org/project/upgrade_status

[~/www/drupal]# composer require --update-no-dev --update-with-dependencies drupal/upgrade_status
Using version ^2.9 for drupal/upgrade_status
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies
. . . 
Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
32 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
[~/www/drupal]# ./drush en upgrade_status
[success] Successfully enabled: upgrade_status
[~/www/drupal]# ./drush cr

Browse to your site, and in the  the Reports menu, select Upgrade status.

You first need to be sure that your hosting site has all the resources you need to actually run Drupal 9.

Your host also needs to allow you 2 GB of php memory of composer will die. If you fail any of these tests, switch your hosting server. See https://jamesrome.net/drupal/hosting

A Note about Composer

Composer 2.0rc is now available. It may solve your problems, or it may cause them. It has several important advantages:

• It is MUCH faster than Composer 1.x
• It uses MUCH LESS memory, to avoid messages like:
  Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 4096 bytes) in phar:///home/mysite/public_html/composer.phar/src/Composer/DependencyResolver/Solver.php on line 223
  and also respects memory limits such as
  In .bash_profile
  export COMPOSER_MEMORY_LIMIT=2G
  alias comp="COMPOSER_MEMORY_LIMIT=-1 /home/mysite/www/composer"

To upgrade to composer 2, do
composer self-update --preview

One of my sites would not allow me to update anything, even from the command line(!), but updating to Composer 2 solved this.

Clone your Drupal 8 site

On your current host, go into the directory above your Drupal 8 root and tar up the public_html directory or its equivalent. Do NOT use zip. It does not preserve permissions!

tar -cvf public_html.host.tgz public.html

Also, using cpanel, select phpMyAdmin, and export your database.

I use my home linux (opensSUSE LEAP 15.2) machine to do my site development.

in /srv/www/htdocs, untar your public_html.host.tgz file:

tar -xvf public_html.host.tgz

it will create a public_html directory with all your files. Also, use phpMyAdmin to install the exported database into your development environment. Use the same database name, user name, and password you used on your hosting server.

I do all my development as root, so the untar will ruin the file ownership, so on my apache2 machine, I must go into /srv/www/htdocs and do

chown -R wwwrun:www public_html

Edit settings.php

If your site was set up properly, you will need to go into sites/default/settings.php and comment out the trusted_host_patterns section. Also be sure your public, private, and sync locations will work on your development machine.

Update or uninstall all your modules

This is the hard part, and it helps to have a big monitor.

If you do not uninstall all D9 incompatible modules from D8, the upgrade will fail, because this is the only way to properly fix your database.

Access your cloned site in a web browser. How you do this depends on your configuration. For example, if you are physically on the development machine, you can use https://localhost/public_html. If your main installation is in the web directory, you may have to add a /web at the end. This also depends on the .htaccess file in your public_html directory.

You will need to open two ssh windows into the public_html directory. In one, open composer.json in a text editor. In your browser, open the Extend/Uninstall Drupal menu.

The first goal is to uninstall all of your unneeded modules.

After each step, it is wise to continue to check that your D8 installation works. The drush updb and cr commands should work with no errors, and your GUI Web page should still work.

You MUST first uninstall each module in the Extend/Uninstall Gui. This gets rid of the module's entries in your database. If you do not do this diligently, you will get all sorts of errors that are impossible (for me) to recover from. Once the module is uninstalled,

composer remove drupal/module_name

will remove the module from the require section of composer.json. In addition, this will fix composer.lock, which wilkl otherwise reinstall removed modules.

It is OK to uninstall multiple modules at once, but the uninstall process might trash things like text formats, so you may also need to edit them to remove references to the removed modules before doing the uninstall.

Gotchas:

• The module file name may not be the thing listed in Extend/Uninstall!
  The module file gdoc_field is listed as Embedded Google Docs Viewer
• You may have to install unwanted modules and then uninstall them in the GUI to properly remove their dependency.
  mailing_list could not be uninstalled until I installed and uninstalled email_confirmer

It is probably good to remove these module files from the modules/contrib directory.

Next, every module must be checked for Drupal 9 compatibility

Go through each line of the require section of composer.json and go to the module site to see if the module has a Drupal 9 version. This is easier said than done. If you are lucky, the release version will have

Requires Drupal: ^8 || ^9

If it says

Requires Drupal: 8.x

look down a bit for a development version. If it is dated more than a year ago, you are probably out of luck. If it is dated since the Covid era, click on it, and you may be surprised to see it is good for Drupal 9. In this case, edit the version in composer.json so you get the dev version. So, for example, replace

"drupal/front": "^1.0@beta",

with

"drupal/front": "^1.x-dev",

There is yet a third possibility: there may be a patch to make the module compatible with D9. So look at the issues and see. However, this poses a problem because you will have to jump through hoops to get composer to manage the module. What I do is remove the module from composer and patch it (if the patch works). Eventually, there will be a release or dev version with the patch included, and I can then add it back into composer.json.

Otherwise, you will need to remove the module following the steps in the unneeded modules section.

Install the updated modules and pray

copy vendor to vendor.sav, and copy composer.json to composer.json.sav

cp -R vendor vendor.sav

composer update --no-dev --with-dependencies

If you are lucky, this will work. BUT, it may not:

composer update --no-dev --with-dependencies
Loading composer repositories with package information
Updating dependencies
{
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package drupal/audiofield ^1.x-dev exists as drupal/audiofield[1.0.0-alpha1, dev-1.x, 1.x-dev, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0] but these are rejected by your constraint.
  Problem 2
    - The requested package drupal/front ^1.x-dev exists as drupal/front[dev-1.x, 1.x-dev, 1.0.0-alpha1, 1.0.0-alpha2, 1.0.0-beta1, dev-9.1.x, 9.1.x-dev] but these are rejected by your constraint.
  Problem 3
    - The requested package drupal/module_filter ^3.x-dev exists as drupal/module_filter[dev-1.x, 1.x-dev, dev-3.x, 3.x-dev, 3.0.0, 3.1.0] but these are rejected by your constraint.
  Problem 4
    - The requested package drupal/uptime_widget ^1.x-dev exists as drupal/uptime_widget[dev-1.x, 1.x-dev, 1.0.0, 1.1.0, dev-2.x, 2.x-dev] but these are rejected by your constraint.
  Problem 5
    - Conclusion: don't install drupal/core 8.9.x-dev
    - Conclusion: remove drupal/core 8.9.5
    - Conclusion: don't install drupal/core 8.9.5
    - Conclusion: don't install drupal/core 8.9.4
    - Conclusion: don't install drupal/core 8.9.3
    - Conclusion: don't install drupal/core 8.9.2
    - Conclusion: don't install drupal/core 8.9.1
    - Conclusion: don't install drupal/core 8.9.0
    - Conclusion: don't install drupal/core 8.8.x-dev
    - Conclusion: don't install drupal/core 8.8.9
    - Conclusion: don't install drupal/core 8.8.8
    - Conclusion: don't install drupal/core 8.8.7
    - Conclusion: don't install drupal/core 8.9.0-rc1
    - Conclusion: don't install drupal/core 8.9.0-beta3
    - Conclusion: don't install drupal/core 8.9.0-beta2
    - Conclusion: don't install drupal/core 8.9.0-beta1
    - Conclusion: don't install drupal/core 8.8.6
    - drupal/module_missing_message_fixer 2.0.0 requires drupal/core ~9 -> satisfiable by drupal/core[9.0.x-dev, 9.1.x-dev].
    - drupal/module_missing_message_fixer 2.0.x-dev requires drupal/core ~9 -> satisfiable by drupal/core[9.0.x-dev, 9.1.x-dev].
    - drupal/module_missing_message_fixer 2.0.1 requires drupal/core ^9 -> satisfiable by drupal/core[9.0.x-dev, 9.1.x-dev].
    - drupal/module_missing_message_fixer 2.0.2 requires drupal/core ~9 -> satisfiable by drupal/core[9.0.x-dev, 9.1.x-dev].
    - Can only install one of: drupal/core[9.0.x-dev, 8.8.5].
    - Can only install one of: drupal/core[9.1.x-dev, 8.8.5].
    - Installation request for drupal/core ^8.8.5 -> satisfiable by drupal/core[8.8.5, 8.8.6, 8.8.7, 8.8.8, 8.8.9, 8.8.x-dev, 8.9.0, 8.9.0-beta1, 8.9.0-beta2, 8.9.0-beta3, 8.9.0-rc1, 8.9.1, 8.9.2, 8.9.3, 8.9.4, 8.9.5, 8.9.x-dev].
    - Installation request for drupal/module_missing_message_fixer ^2.0.0 -> satisfiable by drupal/module_missing_message_fixer[2.0.0, 2.0.x-dev, 2.0.1, 2.0.2].

So, this message tells you that for some reason, there is trouble with 4 modules.

• Try installing the modules individually, e.g.,
  composer require --update-no-dev --update-with-dependencies drupal/module_filter:3.x-dev
• Remove these modules from composer.json and try the update again. Then reinstall the modules one at a time.

Here is another gotcha:

composer require --update-no-dev --update-with-dependencies drupal/smtp
Using version ^1.0@RC for drupal/smtp
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies
Package operations: 2 installs, 0 updates, 0 removals
  - Installing phpmailer/phpmailer (v6.1.7): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing drupal/smtp (1.0.0-rc4): Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup

Note that this installs phpmailer/phpmailer, not drupal/phpmailer. The former is a dependency of smtp (tthe phpmailer library) that is installed inside of vendor. So you will not see phpmailer in your module list (and it is not yet ready for Drupal 9).

If at all possible, you should have a working, Drupal 8.9.x with Drupal9-ready modules, and the drush updb and cr commands should work with no errors. Id you get many conflicting dependence errors, you may have to delete vendor and composer.lock.

Update to Drupal 9

Thus far, all these changes have been performed on Drupal 8. It is possible that you have not completely eliminated all the errors above. For example, one module I needed to update would only work on Drupal 9 (probably an error in the module). But if you do not have any database errors, it is time to upgrade to Drupal 9. It is better to hit your head on the version you really want. Make a copy of composer.json outside of your Drupal installation so you can go back.

Edit the require section of composer.json to require the Drupal 9 files:

    "require": {
        "composer/installers": "^1.0.24",
        "drupal/core-composer-scaffold": "^9.0.5",
        "drupal/core-project-message": "^9.0.5",
        "drupal/core-recommended": "^9.0.5",
        "drush/drush": "^10.3",
        "drupal/core": "^9.0.5",
        ....

9.0.5 was the current core version when I wrote this. Use the version current at your upgrade time. Then perform the update:

composer update --no-dev --with-dependencies

and pray.  When I did this, there were still modules I had to fix (see below). Use the techniques discussed above.

Here is what I had to do on my installation:

composer.json changes

imagemagick    ^3.1
imce    ^2.3
zurb_foundation    ^6.0
audio field    1.x-dev
module_filter    3.x-dev
uptime_widget    1.x-dev
file_entity   (has 9.0 available 'drupal/file_entity:2.x-dev')
devel": "^4.0.0"
front:1.x-dev'
recaptcha ^3.0
entity": "^1.1
module_missing_message_fixer": "^2.0.0

Removed modules
audio_embed_field
eform
video
video_filter
migrate_tools
migrate_upgrade
phpmailer
httpbl
libraries
db_maintenance
context
file_entity
hacked     (has patches)
media_entity_actions
entity_extra_field

to get composer update to work I also removed

audiofield
front
module_filter
uptime_widget

I got more errors:

 [error]   (Currently using Incompatible module The following module is installed, but it is incompatible with Drupal 9.0.5:
 * Insert Block (I removed it)

Review the  suggestions for resolving this incompatibility [1] to repair your
installation, and then re-run update.php.

[1] https://www.drupal.org/docs/8/update/troubleshooting-database-updates
)
 [error]   (Currently using Incompatible theme The following theme is installed, but it is incompatible with Drupal 9.0.5:
 * Showcase Lite

Review the  suggestions for resolving this incompatibility [1] to repair your
installation, and then re-run update.php.

[1] https://www.drupal.org/docs/8/update/troubleshooting-database-updates
)
 [error]   (Currently using Missing or invalid module The following module is marked as installed in the core.extension
configuration, but it is missing:
 * audiofield

I reinstalled audiofield

composer require --update-no-dev --update-with-dependencies drupal/audiofield:1.x-dev