How to improve the creation of tables of code and comments

Here's another possibility. Since your problem is fundamentally a problem of the comments being stripped, we can define an invisible wrapper Commented that evaluates away to nothing when operated on, and formats like a comment.

Here's a possible imp.

First make the formatting right:

Format[Commented[e_, c_]] :=
 RawBoxes@
  TemplateBox[
   {
    ToBoxes[Unevaluated@e],
    ToBoxes[c]
    },
   "CommentedCode",
   DisplayFunction ->
    Function[
     RowBox[{#, " ",
       TemplateBox[{#2}, "Comment",
        DisplayFunction ->
         Function[

          StyleBox[RowBox[{"(*", #, "*)"}], 
           ShowStringCharacters -> False]
          ]
        ]
       }]],
   InterpretationFunction ->
    Function[RowBox[{"Commented", "[", #, ",", #2, "]"}]]
   ]

In[15]:= Commented[a, "test symbol"]

Out[15]= Commented[a, "test symbol"]

But if we look at its format form (i.e. copy it with Shift-Control-C):

a (*test symbol*)

Then make it invisible to evaluation:

Commented /: (h : Except[Hold | HoldForm])[a___, Commented[expr_, _], 
   b___] := h[a, expr, b];

1 + Commented[a, "test symbol"]

1 + a

Then you can define a function that will find all Commented annotations, like so:

extractComments[e_] :=
  Cases[HoldComplete[e],
    Verbatim[Commented][a_, 
     b_] :> (HoldComplete[a] -> b), \[Infinity]];
extractComments~SetAttributes~HoldAllComplete

And here's a chunk of nice formatted code to work from:

chunk = HoldForm@
  Commented[
   Table[
    a~Commented~"Return the int",
    {a, 1, 10}
    ],
   "Create a list of ints"
   ]

Table[a (*Return the int*),{a,1,10}] (*Create a list of ints*)

Then:

With[{c = chunk}, extractComments[c]]

{HoldComplete[a] -> "Return the int", 
 HoldComplete[Table[Commented[a, "Return the int"], {a, 1, 10}]] -> 
  "Create a list of ints"}

You can start to work with (an adapted form of) this data structure now, potentially more easily than before

This answer is for a less general question:

• How to improve the creation of tables of code and comments for monadic pipelines?

As I mentioned in the formulation of the original question post, I am interested in making tables of code and comments in order to explain monadic programming. So, it occurred to me at some point that a special monad can be used to make those tables for monadic pipelines.

(To be clear, the problem gets simplified if we want to build code-comment grids for monad pipelines only.)

The resulting TraceMonad code is fairly simple, and demonstrates well the "programming semicolon" view of the binding operator in monadic programming.

I would say in this case the advice "eat your own dog food" is very useful -- it brings a nice solution (although a specialized one.)

In the example below note that :

1. the tracing is initiated by just using TraceMonadUnit;

2. pipeline functions (actual code) and comments are interleaved;

3. putting a comment string after a pipeline function is optional.

Example

The example below has sparse explanations, but the TraceMonad file has fairly detailed ones.

Load packages:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicTracing.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MaybeMonadCodeGenerator.m"]

Generate Maybe monad code for "Maybe":

GenerateMaybeMonadCode["Maybe"]
GenerateMaybeMonadSpecialCode["Maybe"]

Make up data:

data = {0.61, 0.48, 0.92, 0.90, 0.32, 0.11};

Execute a monadic pipeline and generate a table of code and comments:

TraceMonadUnit[MaybeUnit[data]]⟹"(* lift data into the monad *)"⟹
  MaybeFilter[# > 0.3 &] ⟹"(* filter current value *)"⟹
  MaybeEcho ⟹"(* display current value *)"⟹
  MaybeOption[(Maybe@Map[If[# < 0.4, None, #] &, #] &)]⟹"(* map values that are too small to None *)"⟹
  MaybeEcho ⟹
  TraceMonadEchoGrid[];

This is a problem in design, and the chief difficulty in design is understanding the problem. Suppose you were charged with automating an algorithm in some complex subject area, say finite automata, so your organization could have fairly low-level workers give a it set of inputs and return a nicely formatted correct answer. So, your first action should be to write a set of requirements, preferably with a small set of input data, and a picture or diagram of what the output should look like. Define success.

Next, suppose you are given the finite automata algorithm to use or you find one in a text. You should also find a worked example, a problem at the end of the chapter, or make up a simple set of input data. The algorithm will be a few lines of English text, a few lines of (Boolean) algebra, a few lines of text, etc., etc., ....

For your problem, you should devise a written algorithm or recipe in a few lines of English text to go from simple input data to the desired output.

Now, on one of more sheets of paper, go thru the algorithm one line or sentence at a time to find an answer to your test problem. Written language is VERY imprecise, and it may hours or days before you really understand the algorithm author's (or your own) intent. Do this for several test problems until you really, really understand what the author means or what your problems really are. This can take a long time.

Next, translate your understanding of the algorithm to computer code. In Mathematica, you probably want to use the Module construct (Module[{constants, variables}, instructions, answer]), but you can also put all the initialization steps, constants, variables, instructions, answer in a single cell and just evaluate and re-evaluate that cell.

In these situations, I almost always use the Catch/Throw construct to incrementally arrive at the correct answer:

Catch[Module[{constants, variables},
one or more lines of code
print intermediate result
Throw[intermediate result]
] (* End Module *)

Compare the intermediate result with your worked example.

Repeat
   Write a few lines of code
   Compare result with example
until result is answer to problem

There is nothing worse than computer code the yields the wrong answer. The literature is full of horror stories of people who solved complex problems with Excel, only to find that when the algorithm was used in production, it cost the company thousands, and occasionally millions, of dollars to make it right with the customer. Hence, in Computer Science, Directive 0 is, "If it is not tested, it does not work."

Derive one or more new sets of data to test the solution to the problem. What happens when the inputs become really big, really small, and some big and some small?