If you want to show static relationships between your components, then UML Component Diagram should help you. If you want to show interactions between these components, then try to use UML Sequence Diagram. You should also remember about staying at the same level of abstraction, e.g. don’t go into details (like pressing checkbox), when you’re trying to describe the system on the component level (“gui” vs “other” unit). Your workflow diagram doesn’t have much sense right now, as it should focus on a specific scenario (e.g. loading a report) rather than trying to show all possible scenarios on a single diagram.

Also I think that you have more serious problem underneath, as you “other unit” is just doing too much work, and is clearly breaking the Single Responsibility Principle. You could think about splitting the “other unit” into more components, each of them taking just a specific responsibility (e.g. parsing files, generating reports, sorting reports).

Classic UML Diagram References

Two great references for UML and its many diagram types include Martin Fowler’s UML Distilled in which each diagram is described in much the same way as you might expect a pattern to be described (it is named, described, the forces on it are enumerated, and the applicability is identified). A more comprehensive reference is Grady Booch (et. all) in Object Oriented Analysis and Design with Applications.

Nested State Diagrams

You mention that your workflow looks like hell, and often the art of doing a good diagram is its level of detail. The Booch reference has great material about nested state machines that knocked my socks off when I read it. A state diagram can have a boat load of states and transitions, but by using their nested representation a lot of transitions that can happen from any state can be represented in an outer diagram and the transitions that are most specific to a locality of a few states can be represented by an inner diagram.

Activity Diagrams

Workflow might lend itself to being shown in an activity diagram. Activity diagrams are superior to flow chart in several ways, but can represent some similar things like decisions and iteration. One thing I like about them is that they show parallel activities, and sometimes represent the division of labor in the system using partitions or as they are more commonly called, swim lanes. They can also represent synchronization with a synchronization bar that has one arc coming in and multiple arcs going out (fork) and multiple arcs coming in and a single arc going out (join). Like state machines, UML also permits activity diagrams to be nested to better structure and to limit the level of detail.

Data Flow Diagrams

If you don’t mind a blast from the past, one diagram from structured diagram that UML rejected but for which I believe UML has not given a fully satisfying replacement is the data flow diagram (DFD). This type of diagram is beautifully simple in its use of four symbols to show a process, data store, data flow, and external entities. External entities are show at the right and left sides of the diagram to show who or what provides inputs and outputs to the system, the data is labelled as it is passed from one data transforming process to another. If data is sourced from or produced for storage, that too is shown, typically with the data stores located below the processes that consume/produce them.

Using Diagrams to Design Better

Many other diagrams are available. Your favorite diagram editor might expand or limit your choices. Ideally, diagrams should help you improve the consistency and completeness of your designs. A complex hard to draw diagram might be telling you that your design is too complex, and may even show you some relationships that you can make simpler.

My suggestion would be creating an architecture diagram to illustrate the high level idea of the whole project.

also if class diagram is too complex then break them units and show the interaction of units rather than all classes.