Reverse engineering is inherently a tricky business. Even in the best case, with relatively documented products, APIs, or protocols, there will be a lot of details, edge cases, and most of all, original intent that is not stated.

Undocumented systems come next. You have to sit and guess what’s going on, try experiments, puzzle over the outcomes, and basically Do Science on the system.

You have the worst case. A system that is actively trying to prevent discovery. The developers of your disassembler clearly understood that “hey, let’s try this on itself!” would be some folks’ first thought. And the really bad news: There are a lot of different ways to impede discovery. And by dint of them being a disassembler team, they know a good many of them.

So if you’re really intent on disassembling this, you will need to either (i) get lucky (e.g. by finding “the answer” on the Internet, which you’re already trying), or (ii) be very persistent, tenacious, and disciplined. Reverse engineering is inherently hard, like untangling a tight knot, even in the best case.

1. Start the easiest way: Use a different disassembler. Competitors aren’t likely to care if you disassemble hopper.

2. Hypothesize defenses and experiment with ways to defeat them. One technique they may be using is hashing various byte sequences within their own executable, then looking for such sequences when they disassemble. You may be able to subvert this by divide and conquer: e.g. by creating copies of the executable that are slices out of the original. You will probably need to know what the proper executable format is and conform your slices to that, so that you’re not presenting it an apparently random sequence of bytes, but something that is plausibly an executable. On those pieces you might try varying degrees of fuzzing (i.e. changing random bytes). You will need an automated testing rig for this, because the number of variations you are likely to need to test to get positive results is very large. This would be like decoding the genome by decoding pieces of it at a time.

3. Do what virus-hunters do. Follow the hopper code execution instruction by instruction from its very first instruction executed. This can be done in a machine code simulator, a step-by-step instruction debugger, or by hand. It’s tedious, painstaking, and can require a fair number of tools or a lot of patience. But suss out how the program loads up and establishes its beachhead. This can also be combined with the slicing approach; once you know a snippet of the hopper bootstrap, slice that into a hypothesized short executable and see if hopper recognizes it. If it does, start fuzzing some of the value (either in the executable format/framing, or in the bootstrap sequence).

4. Work, work, work. Reverse engineering is almost always hard. And here you face intentional impediments and (likely) obfuscation. There’s a whole skill set you’ll need to develop, much like debugging. Keep pulling those strings, unraveling a little bit at a time. Good luck!

It appears to be keying off the presence of an internal string, which in my experiment was found to be “(c) 2014 – Cryptic Apps SARL – Dissassembling not allowed.” By changing the first character it allowed disassembly, though I didn’t try any later characters – I stopped after success. This was found by doing an exhaustive binary search for what change allowed it to disassemble itself. A one byte change is all it took.