For all of these items, I know how to do them. I have just been lazy and not doing so; either too lazy to get the art assets or too lazy to bother with custom-made elements instead of things that can be easily duplicated in a standardized way.
To me, Layout is like building a dream in Inception. So listen to this while you read.
Icons
I have not been using icons very much at all in my work so far. I plan to do so going forward. I have purchased some resources in this vein and I have tapped a lot of free ones. I see icons being useful in primarily three ways.
1. Type Divisions: this is when you use icons to designate when certain divisions apply to a particular element. For example, see below.
The muscle flexing icon indicates that the Bad Smell disadvantage (GURPS 4th Ed) is a physical flaw while the head icon indicates that Bad Temper is a mental flaw.
This is merely a simplification of the text to remove clutter. You could add (Physical) to every physical flaw, but that would be cumbersome and clutter-inducing. So you use an icon to simplify it down. The downside is that that if the icons are not immediately obvious in their meaning, someone will have to flip back to reference the icon. You also have to use this sparingly else it can become hard for someone to remember what they all mean.
2. Paragraph Inset: this is when you use icons to denote different sections of text under the same sub-header by some additional kind of meaningful division. For example, see below.
The Davion Icon indicates that all this text (this is at the start of an entire page on Davion in a Battletech book) is about House Davion. There are then subsequent pages with Kurita and so on. The point of these are to serve as visual anchors when you are flipping through the text later, as well as solidify the connection between that information and the concept of House Davion (i.e. make it easier to remember by hooking it to a visual). Usually these kinds of icons have to be truly striking visually to serve their purpose.
3. In conjunction with a demonstrative graphic design: this is when you use the icon as part of a graphic structure to represent game concepts. This is actually very very rare in RPGs, so much so that I don't have a PDF example to draw from that comes from industry. I will have to fall back to my own page from Novarium, posted a few days ago.
As you can see, the point of the icons is not just to identify each element in a textual way, but to allow you to see that you pick one from the upper group, one from the lower group, and create a spell of your own design. This works in conjunction with the support elements of page; the lines and the text boxes.
Boxes
I already have boxes in my games, but I use a basic style. I am going to be improving them. As with all my games, I am going for an open modern uncluttered look. So the below are some examples of a few things that I am going to be trying to emulate in my own work.
1. Box with Icon inset to denote type: below example from Old School Hack
This also incorporates a line-perimeter style with text breaking the line. I will be using some of that too.
2. The Non-Box: this is a box-like division of the text that is... not a real box. See below from a Werewolf: Forsaken product.
This create a strong division in the text, exactly what you are going for with the box concept in the first place, only it isn't actually a box.
Headers
I already have headers in my games, but I use a basic style. I am going to be improving them. I am going to be moving toward a complex multilayered format for chapter breaks and then a less complex internal division style. The below example is from Battletech (obviously) and I am going to be trying to create something similar to this style in my own work (though obviously genre appropriate, this is futuristic).
As you can see, there are three horizontal layers:
1. the top edging which looks like concrete with some battle damage.
2. The red rocky sublater that appears to be beneath the two layers around it
3. The steel layer (which extends to the bottom of the page in this piece)
On top of that, between the two upper laters is a logo layer.
----
Closing:
Learning on you own entails finding things you like, reverse engineering them, and then building to suit your own needs. So look for similar structures to what I have highlighted about in future versions my work.