In-depth look on the Magic Requests Broker
(11/04/99)
A. Algorithms being used to match AppServers & Requests

• When a request is accepted by the Broker
• When an AppServer completes a request
• Notes (for environments with multiple distributed applications)

B. Partitioning during work-group development
C. AppServers in Online mode
D. MGRB.INI: AutoLoad parameter
E. MGRB.INI: ReLoad parameter
F. FAQ
G. Host names resolving
H. Ports usage
I.Multi-broker environments
J. Controlling the deployment architecture programmatically using the broker functions
K. Calling the broker from third party tools
L. Log files
M. Asynchronous requests
 

1. Algorithms being used to match AppServers & Requests

When a request is accepted by the Broker :

1. The Broker scans the list of AppServers that support the requested application, sorted by the entry EnginesPriority in MGRB.ini (if the entry is not being used, the order of AppServers is decided by the Broker).
2. Ignore AppServers in any one of the following conditions :
1. The AppServer is busy– in one of the following situations:

• Currently executing a previous request,
• In Toolkit mode,
• Has another application opened, either running a local program, or set to the non-default value ‘No’ in the entry ‘Application Server Can Change Application’.

1. The AppServer is the same engine that sent the synchronous request (asynchronous request can still be sent to the same engine that called them).

3. Choose an optimized AppServer, according to the following order :
1. The AppServer has the requested application already opened, and the AppServer is not running any local program,
2. The AppServer has the requested application already opened, and the AppServer is running a local program,
3. The AppServer has no opened application,
4. The AppServer has another application opened.

When an AppServer completes a request :

1. Scan the applications of the AppServers in the order of applications as in the MAGIC.ini file of the AppServer (Note : Until v8.01, including, the last application was checked first).
2. For each application, check if there are requests pending for it :
• If yes, extract the requests with higher priority and stop the scan.
• If no, continue to (1.)

Notes (for environments with multiple distributed applications) :

• It is highly recommended to let the Broker make it’s own decisions (based on the described algorithms) rather then allocate engines that will be dedicated to one or few applications (in Magic.ini), since by letting the Broker decide, a much higher degree of Load Balancing is achieved.
• It is recommended to set the majority of the AppServers to have the most requested applications in the 1’st entries in their Magic.ini files (Settings/Applications), but to have, for each application, at least one AppServer in which the application will be positioned 1’st. It is recommended to query the Broker for the number of hits per application (RQ* functions), in order to make these decisions.

2. Partitioning during work-group development

 
 
 
 
The broker has a special option to enable developers to redirect their requests to their own AppServers, by using the [MRB_ENGINE_CLIENTS_MAP] section in MGRB.ini.
This is useful during work-group development (Team Development, Archive Manager), since without explicitly directing requests sent from one developer to its own copy of the application, the developer will have to check-in its change each time, in order to see the change.
For example, when a developer activates a request from its WEB browser, he will most likely prefer to activate its modified copy of the application and not the central copy.
Example:
(In MGRB.ini)
[MRB_ENGINE_CLIENTS_MAP]
eran:eran
Any AppServer that runs on the computer named eran (left to the ‘:’) will accept requests only from clients that run on the computer named eran (right to the ‘:’).
It is also possible to use this option to restrict access to AppServers, for example: eran:eran,yoram - Any AppServer that runs on eran will accept requests only from clients that run on eran or on yoram.
IP addresses can be used only for the clients (for example, eran:88.0.0.84,88.0.0.85).
 
 
3. AppServers in Online mode

 
 
Working in online mode is intended only for development or for workflow applications. Using it for Internet/partitioning has a very bad impact on performance!! In such environment in which the Magic engine is used as a batch tasks server the AppServers must be activated in background mode.
 
 
4. MGRB.INI: AutoLoad parameter

This new parameter (since v8.2) allows the Broker a certain level of automatic AppServers activation.
The parameter takes a simple format: AutoLoad=<exe_entry_name>,<cnt>.
The exe_entry_name is one name that exists either in the [MRB_EXECUTABLES_LIST] or in the [MRB_REMOTE_EXECUTABLES_LIST] sections of the MGRB.INI settings file (for explanations on these 2 sections, refer to the REF.PDF electronic manual).
The Broker will automatically activate the specified exe_entry_name, in one of the following 2 cases:

• A request was submitted by a client, and there are no registered AppServers whatsoever. In this case, the Broker will activate the specified exe_entry_nameimmediately, so that the activated AppServer will still be able to serve the incoming request.
• A request was submitted by a client, and there are registered AppServers. In this case, if no AppServer will become available during the request wait time, the request will fail (with an error message of either "No AppServers" or "AppServers are busy"). If at this point the number of AppServers is smaller than the value specified by the <cnt> parameter, then a new instance of the exe_entry_name will be loaded, to reduce the possibility that the future requests will fail to receive a matching AppServer.

Note: This simple mechanism does not take into consideration the name of the application that the incoming request specified. The Broker simply activates a new instance of the exe_entry_name, regardless of what applications that each AppServer can serve.

5. MGRB.INI: ReLoad parameter

 
 
This new parameter (Planned to be added in v8.2h) allows the Broker another level of automatic AppServers activation.
When this flag is set (ReLoad = Y), the Broker functions as a simple watch dog, meaning that whenever an AppServer that was activated from mgrb.ini crashes (from any reason – OS, Application bug or whatever), the Broker will automatically reload a new instance of the same ExeEntry name listed in mgrb.ini.
 
 
6. FAQ

 
 
Q. How does the MRB choose an AppServer when it receives a request from a client? Does it take into consideration the things like:
• Whether the runtime engine is running on a local computer (i.e., in the same machine as MRB), or a remote computer.
• whether the load (CPU usage) of the executing machine is high or low,
• whether the CPU of the executing machine seems to be slow or fast,
• how many Magic runtime engine is running on the machine,

A. The answers to most of the questions are no. We don't use agents on each computer. However, the EnginesPriority parameter gives partial (and manual) control. The answer to the question: "whether the runtime engine already opens the same application as the request" is yes.
---
Q. Is it possible to assign a different name to the Broker service ?   A. Yes. Use the syntax: mgrqmrb.exe –is=name (e.g.: mgrqmrb –is=dev will create a service named "Magic 8 Broker – dev"). To remove this specific service, use the syntax: mgrqmrb –rs=name.
---
Q. How does the Broker handle a request to terminate AppServers when they are busy serving request?   A. The Broker will send a termination message to such AppServers. The AppServer will be terminated as soon as it completes the request that it was busy serving.   ---
Q. After the requester sends the request to the broker - does it send the request again to the engine when there is an available one, or the Broker does it?
A. Synchronous requests are passed twice - once to the Broker (only the Header of the request), and second time to the AppServer - with the arguments.   Asynchronous requests are passed with the arguments to the Broker in the first place, and then the Broker itself pass them to the AppServers. This is because in Async requests the requesters doesn't wait, so it cannot pass the request to the AppServer (it has no need for that - it doesn't wait for the result anyway). In Sync requests, the Broker was design this way in order to avoid being a bottleneck - if the Broker had to perform also the sync requests, it would’ve spent additional time executing the requests against the AppServers.

7. Host names resolving

 
 
The purpose of this section is to highlight a few vague points regarding the tcp/ip hosts resolving. This section is not intended to describe in detail subjects such as DNS, DHCP or other network issues.
The partitioning architecture of Magic 8 allows the different modules (Broker, Clients, AppServers) to be positioned in different computers. For this purpose, each computer must know the names of the other computers that it needs to access. For example, if the Broker is positioned in one computer (named SRVR_1) and there are 2 AppServers in different computers – SRVR_2 and SRVR_3, then the AppServers identify themselves to the Brokers using these names (SRVR_2 and SRVR_3), and these are the names that the Broker passes on to the clients when the clients send synchronous requests. Hence, each client must know how to resolve the names SRVR_1, SRVR_2 and SRVR_3. The best way to achieve this purpose is DNS (Domain Names Service) or DHCP – a centralized pool of names and their known ip addresses. An alternative way (old fashion yet simple and easy to implement) is using the hosts files. The hosts files mechanism requires that each computer must have an up-to-date copy of the same hosts file.
Note: It is possible to use specific ip address to access the Broker (in mgreq.ini or in the servers table of magic.ini or using the command line requester:
mgrqcmdl –host=<ip address> …).
However, using ip addresses to bypass host name resolving (as described above) do not apply for AppServers, which are identified by their names, and these names will be used whenever the Broker or the requesters need to access them, even if they are in the same computer of the Broker. The clients do not make optimization in this area. Each host name is being resolved as is.
Important note: DHCP should not be mixed with non-DHCP settings. Computers that don’t use DHCP will not know the correct ip address translation, because they use another database (for example, a fixed DNS database).
 
 
 
8. Ports usage

 
 
The Broker publishes one port (in MGRB.INI – BrokerPort), and only this port should be used[Note: To Start the Broker on a specific ip address, use the extended syntax of the BrokerPort entry (in mgrb.ini) - BrokerPort = n.n.n.n/n].
However, the Broker actually uses 6 ports - 5 for the external components, and another one for its own internal purposes. These ports take a form of a range, starting from the specified port number: BrokerPort :-: BrokerPort + 5.
A common problem might occur if the interfacing components of the partitioning architecture (Requests, AppServers) use a different port [or don’t use any port – in this case the default port (2001) will be used]. In particular, the Internet requesters are sensitive to such problems, since their default directory is the default directory of the Web Server, and it might be different from the directory in which the MGREQ.INI file was placed. To resolve such ambiguity problems, a registry key (HKEY_LOCAL_MACHINE/SOFTWARE/MAGIC/MGREQ can be used to specify the directory in which the mgreq.ini for the internet requesters was placed.
 
 
9. Multi-broker environments

Using multi-broker environment can serve 2 goals:

• Separated environments [e.g: one environment (Broker + AppServers) for development and another environment for Deployment]. At this point it’s important to note that it is relatively easy to misuse multi-broker environments by using several brokers for several applications, for administration purposes. This is a wrong behavior. In such cases the Broker turns out to be a pipe, losing it’s ability to perform load balancing. The above administration issues are better answered by using services in the Monitor application – by using a service the Broker refers only to the application pointed out by the service [for example: the function activation <RQLOAD (‘Default Service’,)> will return statistics only regarding the application pointed out by the service ‘Default Service’ in the services table]. Another possibility is to use the command line requester: mgrqcmdl –que=load –app=app1
• Another advantage of multi-broker environments is using Brokers as agents to start AppServers on other machines: Other Brokers can be used to start AppServers on their own machines, using the section [MRB_REMOTE_EXECUTABLES_LIST] in the MGRB.INI of the main broker.

Note: Developers may want to access several brokers on the same machine from different requesters activated by a WebServer. This goal can be achieved only if the Web Server reads the MGREQ.INI from a different directory for each requester (e.g: from the directory assigned to a different alias). Such ability can be used to set different MessagingServer entries in the different MGREQ.INI files and thus enable to access different brokers. In case that the Web Server uses the same directory for all of its aliases, then only one broker can be accessed.
Note: The registry key HKEY_LOCAL_MACHINE/SOFTWARE/MAGIC/MGREQ must not be used in order to enable the scenario described above.

10. Controlling the deployment architecture programmatically using the broker functions

 
 
The RQ functions can be used to customize the deployment environment programmatically. For example, during an application installation process, it is possible to query the user for computer names on which AppServers will be loaded, and this information can be saved into data files, and later used to start the AppServers during the Broker startup.
The Broker MGRQ.INI file can also be edited to provide such customization.
 
 
11. Calling the broker from third party tools

 
 
This is the open API, which is already a part of our installation. It can be used for all 4 types of middle-wares – MGBroker, MQ, DQ and Corba.
 
 
12. Log files & Unknown errors

There are 2 types of logs:

• From MGREQ.INI – displays low level activities – tcp/ip, threads, events, etc.
• From MGRB.INI – displays high level activities of the Broker – initialization, receiving requests, locating AppServers, sending AppServers to requesters, etc. This log is useful to understand if a certain request was accepted by the Broker, what was done with it, etc.

Syntax: Log = name sync level.
For example: Log = req.log y s – log name is req.log, sync is y – the log file is opened & closed for each line, thus making it possible to share the same log file among several modules.
Sync: y – explained above, n – closes the log only upon termination, f – flushes the log to the disk for each line, but without closing the log file each time.
Level: c – customer (less logging), s – support, r – r&d, rf – r&d enhanced (should not be used in mgrb.ini).  
Note: The mgreq.ini is also being used by the Broker, and should be used whenever there is a need to trace the low level activities of the Broker (connect/send/recv..).
The mgreq.ini file is used by all of the modules that take part in the Partitioning architecture of Magic: Requesters, AppServers & the Broker. When each one of these 3 types of modules is started, it searches for the mgreq.ini file. If found, the settings are read.
Some of the settings aren’t relevant to all the modules. For example, the Priority entry is relevant only to requesters. For additional details, refer to the documentation of Magic 8 (chapter 15 of the Reference Guide).   Unexpected errors: Errors that aren’t listed in the documentation (e.g. –145, -105) are best handled by using the log file from mgreq.ini. This log file can show for example problems with host name resolving, which mostly cause these errors to be displayed.
It is important to know the difference between errors from the Web Server (which aren’t displayed in the blue/aqua error page of Magic) to errors of Magic. Errors of the Web Server can be a missing requester, insufficient rights, etc.

13. Asynchronous requests

Asynchronous requests are requests that are sent from:

1. Magic engines – call remote : wait = No
2. The command line requester – mgrqcmdl –app=x –prg=y –nowait

The Broker performs asynchronous requests. When a requester calls the async request, the Broker returns an immediate response that contains the message id. When the Broker locates an available AppServer, it sends the request to this AppServer.
 
 
Important note: If AppServer are available to serve async requests, and yet there is a delay until the request reaches the AppServer for execution, it is most likely to be a case of host name resolving (see paragraph 7th), meaning that the Broker cannot resolve the host name of the AppServer. It is the administrator responsibility to verify that the host name of the AppServers are known on the computer on which the Broker was activated.