Some years ago, I was pointed to this wonderful blog post (please excuse the profanity) by Michael Orlitzky. While it is blatantly eccentric, this article has since been perpetually at the back of my mind, because the message it conveys is very bold, makes some great points, and is portrayed in a humorous manner. The idea is that people tend to be too afraid and/or too lazy to either build or use proper package managers. For instance, the trend these days it to make language-specific package managers (pip, cargo, cabal, node, …), platform-dependent package managers (apt), and package managers that only know how to handle binaries (apt, pacman). And this is all garbage, especially the first two categories. With language-specific package managers, users end up with a boatload of disjoint package managers that they almost surely are not keeping up to date, and by construction they do not cohesively manage dependencies. Consequently people use things like virtualenv and conda, which basically says "rather than worry about dependencies, why not just put each project in an isolated box with its own dependencies". This is stupid because now you have several copies of all these dependencies and you're still not managing dependencies properly, in fact you're neglecting the issue of dependency hell altogether. And Docker containers are basically the same idea, but rather than bundling some python code with some libraries, it's an improvement since it bundles an entire operating system! Let me make it very clear that the previous sentence was sarcastic, because this is an absolutely atrocious idea for similar reasons (for the record, I recognize that Docker does make sense in some settings, it just doesn't make any sense to depend on it for projects you're building on your PC).

The author's suggested solution is pretty bold. He claims (correctly) that the Gentoo Linux package manager, portage, satisfies all of the desired properties of package managers. Moreover, since it's cross-platform, he says every software developer should distribute software for portage. Note that the author is a Gentoo developer, so there is clearly some bias. But the more I think about it, the more it starts to make sense to me. And my journey to install ROCm only cemented this idea further.

So in the previous section, I mentioned how the ability for a package manager to build software from source is very desirable. Until I started building software from source frequently, I didn't realize that lots of software is distributed with lots of configurable parameters that actually affect the binary that is produced. For instance, Vim provides a massive selection of features, many of which I have no use for. By configuring the build, I can choose to simply not build in these features, resulting in a more lightweight binary. More severely, building ROCm (and the PyTorch with ROCm support) allow you to configure the build depending on your GPU architecture. Quoting Michael Orlitzky's article, by simply downloading a binary package, you are assuming that whoever built it for you has to

1. know that your architecture exists; and
2. give a s*** about you; and
3. find a computer like yours to test it

Usually one of these criteria is not met (guess which one).

Indeed, at least one of these criteria was not met when I tried ROCm with those pre-built Docker containers, as the ROCm system was built without support for my GPU. Also containers suck, as I explained before, so I set out to build ROCm myself. However, since it's expected that everyone will use pre-made containers, very little effort was made to make the process of building from source a smooth one.

So what's the solution to this problem? Michael Orlitzky suggests that all software developers should distribute software as ebuilds, which are essentially shell scripts that tell Gentoo's package manager portage how to configure the source, build the software, and manage its dependencies. And after this experience, I have to say that I kinda agree with this notion. I mean, I acknowledge that this is an unrealistic proposal, but there are some absolutely awesome advantages to using portage that are just intrinsically great, and I can't imagine any cogent argument that suggests otherwise. Here's some reasons why:

1. portage goes beyond just managing dependencies, it can even figure out dependency satisfaction due to those build configuration options I described above.
2. Since the dependencies are managed so nicely, the ebuilds themselves make up a build guide – that is, you don't need any instructions beyond the ebuilds to get the build up and running.
3. The Gentoo developers have developed useful standards for implementing a vast array of common operations, which significantly reduces boilerplate and requisite mental capacity.
4. The Gentoo developers have also developed automated QA to reinforce good practices. This includes a standard for installing binaries, headers, libraries, and anything else; this alone should substantially reduce build failures.

This list is not comprehensive, by the way.

Thankfully, I've been using Gentoo for a few year already – I didn't install it for the sole purpose of getting ROCm to work properly and to write this post. But you don't actually need to install Gentoo to use portage (it has cross-platform support!), rather, you might prefer simply using Gentoo Prefix. While this does in fact add a package manager to your system, it is at least an extremely robust package manager that can suit all of your needs.

In order for Portage to "register" a third party set of packages (or ebuilds, more precisely), you'll need to set up an overlay. Gentoo provides a nice tool called layman to help with this, however I find it's actually easier to set it up manually. I'll be referring to the variable ${EPREFIX} to account for a Prefix setup as opposed to an actual Gentoo system. On Gentoo, ${EPREFIX} simply refers to the root directory. With that out of the way, let's begin.

Start by cloning my repo and putting it in a nice location. Also make yourself part of the portage group to make editing ebuilds much easier.

$ sudo usermod -a -G portage ${LOGNAME}
$ cd ${EPREFIX}/var/db/repos
$ git clone https://github.com/harwiltz/rocm
$ chown -R ${LOGNAME}:portage rocm

[rocm]
location = ${EPREFIX}/var/db/repos/rocm

Now the overlay is ready to go.

I guess this part is optional if you declare the environment variables manually when installing amd-rocm-meta. But I think this is a nicer solution.

...
HIP_PLATFORM=rocclr
MAKEOPTS="-j4"

The MAKEOPTS variable determines how many parallel processes can be run during the build, people often say to set this to n_cpu_cores + 1. I tend to set it a little lower than that so I can use my machine while building stuff in the background.

Now, before packages are tested thoroughly they are keyworded by the CPU architectures for which they're still being tested. We need to tell Portage to accept these packages. Usually I just run my emerge commands and update my configuration as it tells me to, because it could be a little difficult/tedious to figure them all out a priori. But here's the relevant configuration that works for me.

=sci-libs/pytorch-1.6* ~amd64
=dev-libs/half-2.1.0 ~amd64
=dev-libs/rccl-3.5.0 ~amd64
=dev-libs/rocclr-3.5.0 ~amd64
=dev-libs/rocm-comgr-3.5.0 ~amd64
=dev-libs/rocm-device-libs-3.5.0 ~amd64
=dev-libs/rocm-opencl-runtime-3.5.0 ~amd64
=dev-libs/rocm-smi-lib-3.5.0 ~amd64
=dev-libs/rocr-runtime-3.5.0 ~amd64
=dev-libs/roct-thunk-interface-3.5.0 ~amd64
=dev-util/amd-rocm-meta-3.5.1 ~amd64
=dev-util/rocminfo-3.5.0 ~amd64
=dev-util/rocm-clang-ocl-3.5.0 ~amd64
=dev-util/rocm-cmake-3.5.0 ~amd64
=dev-util/rocm-smi-3.5.0 ~amd64
=dev-util/roctracer-3.5.0 ~amd64
=sci-libs/hipBLAS-3.5.0 ~amd64
=sci-libs/hipCUB-3.5.0 ~amd64
=sci-libs/hipSPARSE-3.5.0 ~amd64
=sci-libs/miopen-3.5.0 ~amd64
=sci-libs/rocFFT-3.5.0 ~amd64
=sci-libs/rocBLAS-3.5.0 ~amd64
=sci-libs/rocPRIM-3.5.1 ~amd64
=sci-libs/rocRAND-3.5.0 ~amd64
=sci-libs/rocSOLVER-3.5.0 ~amd64
=sci-libs/rocSPARSE-3.5.0 ~amd64
=sci-libs/rocThrust-3.5.0 ~amd64
=sys-devel/hcc-3.3.0 ~amd64 # Deprecated, no newer releases
=sys-devel/hip-3.5.1 ~amd64
=sys-devel/llvm-roc-3.5.1 ~amd64
dev-python/CppHeaderParser ~amd64

Note that the file above can be named arbitrarily (as long as it's under ${EPREFIX}/etc/portage/package.accept_keywords).

Finally, if you plan on installing deep learning libraries, I recommend also adding the following,

sci-libs/amd-rocm-meta deeplearning

Like the previous example, this file can be named arbitrarily as long as it's in the right directory. Here, we are setting the deeplearning USE flag to the affirmative for the amd-rocm-meta package, which will activate some more dependencies.

Now we're ready to start the build!

We can now run the command I foreshadowed earlier,

$ sudo emerge -av amd-rocm-meta

When this build completes, you'll be all set to install PyTorch or Tensorflow. I believe these are the only two major ML frameworks that have support for ROCm, although to be honest I haven't looked very far.

As a quick sanity check, execute the following,

$ ${EPREFIX}/usr/lib/rocm_agent_enumerator
gfx000
gfx803

This should list codes that identify you're GPU architecture. The gfx000 output corresponds to a CPU, so if that's your only output, something went wrong.

You can also try running ${EPREFIX}/usr/bin/rocminfo and check if it lists correct information about your GPU.