Examples of Magic Data when Listing Pages

Some examples to try in the symbol tester for listing pages.

When learning Magic Data, it often helps to build up sets of symbols piece by piece and evaluate each part and also conduct some experiments on the way.

List a number of pages beneath a particular page in sitemap order.

SET 138 AS_PAGE LIST_PAGES 10

Sets the starting point as page 138 and lists the first 10 pages in the sitemap below it. To get the pages in the order they were added to the site.

SET 138 AS_PAGE LIST_RECENT_PAGES 10 

Changing the above to list recent pages beneath the current page.

PAGE LIST_RECENT_PAGES 10 

If you want more pages, just increase the number '10' to whatever maximum, though bear in mind the list could get long and the processing consequently hard work.

Suppose we want to only list the pages if they were published today. First we need a cutoff time.

SET 00:01 TIME_TODAY

To avoid needing to re-calculate this, we can save it to a memory. 

SET 00:01 TIME_TODAY SAVE "cutoff_time"

Now we can start building from this with our list of recent pages. There are no further easy steps, we need to put it all together.

SET 00:01 TIME_TODAY SAVE "cutoff_time"
SET 138 AS_PAGE LIST_RECENT_PAGES 10 
FILTER_LIST 
  TIME_PUBLIC GT ( SET "cutoff_time" RESTORE )
END_FILTER

In many cases, SET is optional and will be assumed for a text string, so the above could also be written as:

00:01 TIME_TODAY SAVE "cutoff_time"
138 AS_PAGE LIST_RECENT_PAGES 10 
FILTER_LIST 
  TIME_PUBLIC GT ("cutoff_time" RESTORE )
END_FILTER

Now for some explanation.

  1. Makes a note of the cutoff time.
  2. Lists the 10 most recent pages beneath page 138
  3. Starts a filter. The filter will iterate through each page listed and reduce the list to those where the filter content evaluates to 'true'
  4. Gets the time a page was published and then retrieves the cutoff time previously saved and compares the two.
  5. Marks the end of the filter

At the end of the above, we have a list of pages beneath page 138 that were updated today. We could format that list into a list of page links.

SET 00:01 TIME_TODAY SAVE "cutoff_time"
SET 138 AS_PAGE LIST_RECENT_PAGES 10 
FILTER_LIST 
  TIME_PUBLIC GT ( SET "cutoff_time" RESTORE )
END_FILTER
APPLY_EACH
  PAGE_LINK
END_APPLY_EACH
HTML_OL

The additional lines 6..9 work on each element of the resulting list with APPLY_EACH, create a link to the page, and then turn that in to an HTML ordered list.

Or you could count the number of new pages today.

SET 00:01 TIME_TODAY SAVE "cutoff_time"
SET 138 AS_PAGE LIST_RECENT_PAGES 10 
FILTER_LIST 
  TIME_PUBLIC GT ( SET "cutoff_time" RESTORE )
END_FILTER
COUNT_LIST

Another consideration is what to do with an empty result, where there are no new pages. This can be handled with ZERO_AS_THEN_END to return a message and stop.

SET 00:01 TIME_TODAY SAVE "cutoff_time"
SET 138 AS_PAGE LIST_RECENT_PAGES 10 
FILTER_LIST 
  TIME_PUBLIC GT ( SET "cutoff_time" RESTORE )
END_FILTER
ZERO_AS_THEN_END "No new pages so far today"
APPLY_EACH
  PAGE_LINK
END_APPLY_EACH
HTML_OL

If you only want to list new pages of a particular type, you can use:

SET 138 AS_PAGE LIST_RECENT_PAGES_OF_TYPE "My Page Type" 10 

Rather than use a page id at the start, you could give a path to make it more readable.

SET "/blog" AS_PAGE LIST_RECENT_PAGES_OF_TYPE "Blog Entry" 10

This starts with /blog as the page at the top of the blog area and lists the 10 most recent blog entries.

Suppose you want to list all pages beneath a page, and all their sub-pages as well? That requires a list of lists. First, we list all the children, then we process that list to list all the grand children.

SET 1 LIST_RECENT_PAGES 50 SAVE "children_list"
APPLY_EACH
  LIST_RECENT_PAGES 50 MERGE_INTO "grand_children_list"
END_APPLY_EACH
"children_list" RESTORE MERGE ( "grand_children_list" RESTORE ) REDUCE 
APPLY_EACH
  PAGE_LINK
END_APPLY_EACH
HTML_OL

Here we start by listing up to 50 pages immediately beneath the home page and save that as "children_list". Still using that list, for each page we then list up to 50 pages beneath each and start accumulating a "grand_children_list".

On line 5 we then restore the "children_list" and merge it with the restored "grand_children_list", then for good measure we reduce the list, just in case there are any duplicates or null entries. There should not be, but you never know.

Now we have a our long list, we need to do something with it. Perhaps just print a list of links - a sort of sitemap list.

Or we could combine it with our previous work for all pages changed since midnight.

SET 00:01 TIME_TODAY SAVE "cutoff_time"
SET 1 LIST_RECENT_PAGES 50 SAVE "children_list"
APPLY_EACH
  LIST_RECENT_PAGES 50 MERGE_INTO "grand_children_list"
END_APPLY_EACH
"children_list" RESTORE MERGE ( "grand_children_list" RESTORE ) REDUCE 
FILTER_LIST 
  TIME_PUBLIC GT ( "cutoff_time" RESTORE )
END_FILTER
ZERO_AS_THEN_END "No new pages so far today"
APPLY_EACH
  PAGE_LINK
END_APPLY_EACH
HTML_OL

As another embellishment, perhaps rather than pages added, a more interesting list would be pages changed. The best way to do this is with a symbol provided by the Last Updated addon. A catch here is that the LAST_UPDATED symbol requires a page area (or areas) to work from. So for the home page:

SET 1 LAST_UPDATED "Main"

It also returns a date rather than a time, which we can get round with STR_TO_TIME.

SET 1 LAST_UPDATED "Main" STR_TO_TIME

Putting this together into our overall example.

SET 00:01 TIME_TODAY SAVE "cutoff_time"
SET 1 LIST_RECENT_PAGES 50 SAVE "children_list"
APPLY_EACH
  LIST_RECENT_PAGES 50 MERGE_INTO "grand_children_list"
END_APPLY_EACH
"children_list" RESTORE MERGE ( "grand_children_list" RESTORE ) REDUCE 
FILTER_LIST 
  LAST_UPDATED "Main" STR_TO_TIME GT ( "cutoff_time" RESTORE )
END_FILTER
ZERO_AS_THEN_END "No new or updated pages so far today"
APPLY_EACH
  PAGE_LINK
END_APPLY_EACH
HTML_OL

Because we have built a list of lists, this may no longer be sorted into strict update order. We solve that by sorting the list. The SORT_BY symbol can be used to return sort order value for each item, and these values are then used to sort the list we are working on.

SET 00:01 TIME_TODAY SAVE "cutoff_time"
SET 1 LIST_RECENT_PAGES 50 SAVE "children_list"
APPLY_EACH
  LIST_RECENT_PAGES 50 MERGE_INTO "grand_children_list"
END_APPLY_EACH
"children_list" RESTORE MERGE ( "grand_children_list" RESTORE ) REDUCE 
FILTER_LIST 
  LAST_UPDATED "Main" STR_TO_TIME GT ( "cutoff_time" RESTORE )
END_FILTER
ZERO_AS_THEN_END "No new pages so far today"
SORT_BY
  LAST_UPDATED "Main" STR_TO_TIME
END_SORT_BY
APPLY_EACH
  PAGE_LINK
END_APPLY_EACH
HTML_OL

The above gives pages added or modified today, in the order they were modified. But perhaps we want the most recent first. In that case, we can use REVERSE to list the other way round.

SET 00:01 TIME_TODAY SAVE "cutoff_time"
SET 1 LIST_RECENT_PAGES 50 SAVE "children_list"
APPLY_EACH
  LIST_RECENT_PAGES 50 MERGE_INTO "grand_children_list"
END_APPLY_EACH
"children_list" RESTORE MERGE ( "grand_children_list" RESTORE ) REDUCE 
FILTER_LIST 
  LAST_UPDATED "Main" STR_TO_TIME GT ( "cutoff_time" RESTORE )
END_FILTER
ZERO_AS_THEN_END "No new pages so far today"
SORT_BY
  LAST_UPDATED "Main" STR_TO_TIME
END_SORT_BY
REVERSE
APPLY_EACH
  PAGE_LINK
END_APPLY_EACH
HTML_OL

If all we want is a count of pages changed, we can get rid of the latter part and simply count the pages in the list.

00:01 TIME_TODAY SAVE "cutoff_time"
1 LIST_RECENT_PAGES 50 SAVE "children_list"
APPLY_EACH
  LIST_RECENT_PAGES 50 MERGE_INTO "grand_children_list"
END_APPLY_EACH
"children_list" RESTORE MERGE ( "grand_children_list" RESTORE ) REDUCE 
FILTER_LIST 
  LAST_UPDATED "Main" STR_TO_TIME GT ( "cutoff_time" RESTORE )
END_FILTER
ZERO_AS_THEN_END "No new pages so far today"
COUNT_LIST

Having tested and developed from any of the above using the symbol tester, to use the expression in a content block you will need to reformat to just one line within the [% ... %].

In an html block you can keep the lines to make it easier to read/maintain.

All the above Magic Data involves a lot of processing. If performance becomes an issue, an option would be to write some finely crafted php within a custom symbol.

Or you could have a look at Uber List which provides some more powerful listing and filtering symbols such as LIST_ALL_PAGES_WITH_FILTER and AND_WITH_FILTER to add multiple filter conditions to the list. Expressions combining these symbols are built into optimised database queries for making complex lists quickly.

Last updated: over a year ago