API Clients and Servers
Black Magic Data provides an API that enables pairs of sites to communicate using Magic Data. Client and server sites are defined in the dashboard, then the client site can use API_EVALUATE and API_EVALUATE_END to wrap sets of symbols to evaluate on the server site.
In the dashboard at Dashboard > Magic Data > API Clients and Servers, you can use the Clients tab to list and configure clients - other sites that can use the Magic Data API to evaluate expressions on this site, and servers - other sites that this site can send Magic Data expressions to for evaluation.
The details of a client defined on one site need to be paired up with the details of a server defined on another site. Currently these interfaces need to be paired 1:1. Future updates of the BMD API may include 1:Many interfaces and even public interfaces. However, for now we are keeping the pairing tight to enhance security.
On a site providing an API server, you need to define client sites.
Client sites are defined by site name, url and optionally an IP address filter. All clients are listed in a summary, then an individual client can be edited or new client added.
Here we see the edit dialog in more detail. Note how the client can only evaluate symbols in the permission set 'No Black Magic' on this server. The user is defaulted to 'Guest', but any user could be selected (and given concrete5 user permissions) the API limited to the pages and files visible to a client.
API calls and returns are encrypted using the API key. An API key can be generated automatically on the server, or entered as any text you like. Unless you have good reason to do otherwise, it is recommended you use the automated key generation. You then need to copy that key from the server site's API client edit form and paste that key to the equivalent API server edit form on the client site.
On a site that is a using the API to evaluate Magic Data on API server sites, you need to define Server sites.
The setup is easier than for clients. All you need to enter are the server site name and URL, then paste in the API key copied from the server site.
An information (i) button is provided in both client and server lists to show the information needed for the paired site.
Evaluating Magic Data on an API server
When a client site is configured with interfaces to one or more API servers, the symbols API_EVALUATE and API_EVALUATE_END can be used within Magic Data expressions to evaluate subexpressions on API servers.
LIST_API_SERVERS 10 APPLY_EACH WITH_API_SERVER SET 1 API_EVALUATE AS_USER USERNAME API_EVALUATE_END . ' on ' . ( CURRENT_API_SERVER ) END_APPLY_EACH HTML_UL
Going through the above line by line:
- The client site lists up to 10 servers it can connect to
- For each of those listed servers
- Configure the following API evaluation to use that server
- Set the user id to 1 (the super administrator)
- Make an API call to the server
(the next line is passed to the server for evaluation)
- On the server, find the user and get their username.
- The result is passed back to the client.
- Some text manipulation to concatenate the server name to our output
- Repeat for the next server
- Finish by presenting our list of administrator and server names.
This also illustrates that we set the user id we want to name on the client. That is passed to the server as the initial value (previous result) by the API_EVALUATE symbol. We didn't really need to do that for such a simple expression, but passing a starting value to the server can be useful for more sophisticated applications. Here is the result of evaluating the above in the symbol tester on one of my sites (you can see more detail by popping up the screenshot gallery to the right).
To keep the amount of data down, the API_EVALUATE symbol only passes the previous result value to the server and API_EVALUATE_END only returns the immediate outcome of the symbols evaluated on the server.
For more complicated interactions, you can alternatively use the symbol API_EVALUATE_EXTENDED to copy magic data memories from the client to the server, then copy them back again at API_EVALUATE_END.
Every block of symbols in an API_EVALUATE .. API_EVALUATE_END subexpression involves an http post request from the client to the server. If you do a lot of that in the middle of a big loop, it will have both a bad impact on your page load times and potentially create a big overhead on the server.
Within any page, keep the number of API_EVALUATE .. API_EVALUATE_END sections to a minimum. Where possible, make one API_EVALUATE .. API_EVALUATE_END block assemble a batch of various subexpressions all in one go and use memories to pass their results back all in one go.
Another potential performance issue comes with evaluating rules for Symbol Permissions associated with a client. These are defined on the server and for speed, it is best to have as few rules as possible in any one set. For example, its much faster to list a many packages or symbols in one rule than it is to have separate rules for every package or symbol.
If you have Uber List installed, you can use Uber List's cache mechanism to cache API calls. For example, the above could be cached to only retrieve new information once each day by wrapping the API call within a cahed subexpression.
LIST_API_SERVERS 10 APPLY_EACH WITH_API_SERVER SET 1 CACHE "global" 86400 API_EVALUATE AS_USER USERNAME API_EVALUATE_END END_CACHE . ' on ' . ( CURRENT_API_SERVER ) END_APPLY_EACH HTML_UL
Or even the entire loop of remote API calls, or the entire expression.
CACHE "global" 86400 LIST_API_SERVERS 10 APPLY_EACH WITH_API_SERVER SET 1 API_EVALUATE AS_USER USERNAME API_EVALUATE_END . ' on ' . ( CURRENT_API_SERVER ) END_APPLY_EACH HTML_UL END_CACHE
Within and API_EVALUATE .. API_EVALUATE_END subexpression, the magic data snippets available are those defined on the server, not those defined on the client making the API call.
Snippets evaluate with the server's own symbol permission set, as selected in the Symbol Settings dashboard page. If a snippet contains magic data that should not be available to an API client, you can exclude that snippet when setting up the permission set for the interface.
For a very tightly controlled interface, you could also take advantage of the opposite, exclude All symbols, then specifically enable only those symbols needed to evaluate snippets where you have pre-scripted what a client site can do. With this approach you can create powerful interfaces as snippets, but prevent an API client site from doing anything provided by those snippets.
Faults and Fault Handling
If a server fails to respond, the API evaluation on a client will timeout. You can configure this timeout via the Symbol Permissions dashboard page and it is defaulted to 10 seconds.
Following a timeout, the API_EVALUATE_END symbol will return null. Your magic data expression on the client should be able to cope with that, either by being immune to a null return or by specifically testing for a null return from the server, rather than failing completely.
There is no protection within API_EVALUATE subexpressions to prevent further nested API_EVALUATE subexpressions. Long chains of such could easily cause server timeouts at the start of the chain. Even worse, if a chain ended up in a recursive loop, where servers pass sub-expression evaluation round between them, you could lock up all servers in that loop. So please take extreme care if you really must nest API_EVALUATE expressions.
If you do create such a loop, the only way to stop it is to hope you can get into the dashboard Symbol Settings page and disable magic data.
By default, API calls are made using CURL. CURL is supported by most servers and is the preferred interface. However, sometimes a php installation will not have CURL available, or it may be buggy. Alternative methods can be selected in the Symbol Settings page.
IP Address Filtering
As a basic security mechanism, An API server can configure a client interface to only be available to a speficic IP address. Sometimes a client server could have an IP address that varies within a pool of addresses. Any section of an IP address can contain a * for any match, or nnn-mmm for a range match.
- 126.96.36.199 - only accept API clients from this address
- 104.25.24-25.33 - accept API clients with third section in the range 24-25
- 104.25.*.33 - accept anything from 0 to 255 in the third section of the IP address
IP address filtering is not very thorough. An attacker could spoof a known IP address. However, for the majority of use it has the advantage of being quick.
If you really want speed, you can also set up IP address filtering in an .htaccess file.
For stronger security, the Black Magic Data API is encrypted.
Encryption is adapted from the core encryption helper. The API key is used to encrypt communication from the client to the server. Within each communication, the client generates a new key that is passed to the server used to encrypt the data returned from the server to the client.
The Black Magic Data API is currently restricted to 1:1 pairing between concrete5 sites. Both client and server sites need to have Black Magic Data installed.
If a sever needs to support multiple clients, rows on the API client list can be copied for each client. Similarly for a client that needs to access multiple servers. But this is a manual process. The API is convenient for a few clients, but not for hundreds or thousands.
Future enhancement may add capabilities for:
- 1:Many server to client interfaces
- Unencrypted interfaces
- Alternative encryption methods
- Public interfaces
- Automated API key distribution from server to client
- Automated API setup in response to client requests
Last updated: over a year ago