RubyGems is a package manager for the Ruby programming language that provides a standard format for distributing Ruby programs and libraries.
Homebrew is a free and open-source software package management system that simplifies the installation of software on Apple’s operating system, macOS, as well as Linux.
Note: Homebrew also can build/install/manage different software packages including gems.
About Ruby Version Manager(RVM):
RVM is a command-line tool which allows you to easily install, manage, and work with multiple ruby environments from interpreters to sets of gems.
What is the difference between ruby gems and brew formulas?
RubyGems and Homebrew are both package managers. RubyGems was created for the express purpose of installing gems whereas Homebrew is a more general tool which can build, install and manage different software packages including gems.
Homebrew is tied to Mac OS X, whereas RubyGems works on multiple platforms.
Best advice? Do whatever the book/tutorial you’re following tells you to do and let your preferences develop over time until you have enough experience to know what you want. All this having been said, it seems as though you’re going to be working in Rails, so you’re going to be using bundler, rather than RubyGems or Homebrew.