What is the difference between the different package files?

edit

The package.xml is the ROS package manifest format ratified by REPs (different ones for each format version: REP 127 format 1, REP 140 format 2, REP 149 format 3. Every ROS package should have one of those. When being released with bloom the information from this file are being used to build the Debian control / rule files.

A colcon.pkg (docs) file must be placed in the root of a package (could be a ROS package, a plain CMake project / Python package, or anything else supported by colcon via extensions). It allows to specify package specific arguments. You would choose that is adding a file to the sources is feasible and you want to alter / amend any kind of meta data as well as custom command line arguments. If a package already has a ROS manifest this file can still be used to add additional information, e.g. about scripts which should be sources in order to provide completion (see ros2cli).

meta files (docs) allow you to provide meta data for a package without placing the file in the source. They are mapped to a specific package based on their name or relative / absolute path. The meta data can contain information like additional dependencies (commonly when the package doesn't have a package.xml file) or information about hooks. E.g. for Gazebo there is a .meta referencing its setup script so that it gets sourced automatically when you source the setup file in the workspace level. Another example are CMake flags for FastRTPS to enable security by default.