On any of the ARM64 chips (currently M1, M2 or M3), you need to link to libraries that support that architecture. CRAN has made those libraries and tools available; see https://mac.r-project.org/tools/ for downloads. They recommend that you install these tools in /opt/R/arm64.

If you do that, you should also put /opt/R/arm64/bin in first place on your search path, because some of the executables in that directory help rgl to configure itself for your system. For Intel Macs and other Unix-alikes, the tools would typically be put into /usr/local, and you may still have some of those in place if you have upgraded from another machine.

Unfortunately, the current RStudio version 2024.04.2+764 automatically puts /usr/local/bin at the head of the search path whenever R starts. This can cause rgl builds to fail.

There are several possible solutions:

1. Don’t use RStudio to install rgl. You can install it by downloading the tarball and then from the terminal running

   R CMD INSTALL rgl_x.y.z.tar.gz
   

   where x.y.z is the version number you are installing. As long as you have your PATH environment variable set up properly, that should work.

2. Within RStudio, you can change the PATH by getting the old one, then setting a new one. For example,

   Sys.setenv(PATH=paste0("/opt/R/arm64/bin:", Sys.getenv("PATH"))
   

   This adds a possibly duplicate copy of the proper directory at the start, and should allow you to build rgl from source and install it. The problem with this approach is that it has to be done every time before installing rgl, because this version of RStudio will mess up the PATH again after the install.

3. I don’t recommend this one, but it’s possible: find every executable file that is in both /usr/local/bin and /opt/R/arm64/bin, and delete the copies in /usr/local/bin. If you do this, it won’t matter which order those two directories are listed in the PATH, it will still work. However, this is error-prone, and you might end up deleting something that shouldn’t be deleted. If you do it properly, then you should be able to build in RStudio without worrying about what it did to your PATH.