AQL Editor

The AQL Editor provides a framework to develop and test AQL files. Annotation Query Language (AQL) is the language in the InfoSphere® BigInsights™ Text Analytics component for building extractors. An extractor , sometimes referred to as annotator, is an information extraction program that extracts structured information from unstructured or semistructured text by using AQL constructs. AQL is the primary method of creating new extractors in InfoSphere BigInsights Text Analytics.

By using the AQL Editor, you can create an AQL file or script and develop the code to perform Text Analytics. An AQL script contains AQL statements. An extractor can consist of one or more AQL files and associated dictionaries and UDF JAR files.

Compilation

You must compile AQL files before you run them. To ensure that the AQL compiler is started whenever you save an AQL file, select the Build Automatically option from the Project menu.

The AQL code of the extractor is compiled by the Text Analytics Compiler into Text Analytics modules (TAM files). A Text Analytics Module (TAM) file is a binary representation of the extractor and contains the execution plan of the extractor, the module metadata, and the compiled dictionaries. Non-modular AQL files are compiled into a genericModule.tam, and modular AQL files are compiled into a file called <moduleName>.tam

In its source form, an AQL module is a collection of AQL source files that are grouped under a folder whose name is identical to the name of the module. Optionally, a module can also contain dictionary files and binary UDF JAR files. A compiled AQL module contains the compiled operator graph, compiled dictionaries (if any), UDF JAR files (if any), and module metadata. AQL modules facilitate modularity and reusability of AQL code. AQL modules help build reusable text extractors that can be used as building blocks to compose complex Text Analytics applications.

Problems view

The Problems view displays errors and warnings with your AQL code. If an error or warning occurs during compilation, it is displayed in the Problems view with detailed information. If you double-click the error or warning, the AQL Editor shows the specific line in the AQL file where it occurred. Errors and warnings are also identified on the marker bar of that particular AQL file in the AQL Editor.

For projects that use non-modular AQL code, if the error or warning occurs on the main AQL file or an included file that is present in the current project, the exact file and line number is highlighted. However, if it occurs in an included file that is not a part of the workspace or is a part of a different project in the workspace, then it is highlighted in the main AQL file.

For projects that use modular AQL code, errors and warnings in an AQL file are marked in the file at the lines where it occurred. However, in certain cases it might not contain associated file information. An error or warning of this type is marked against the project that contains the AQL code that is being compiled. The message contains the exact details and line number of the problem file.

The following information is displayed in the Problems view:
Table 1. Problems view information
Column Definition
Description The compiler error description and suggestions for resolving the error.
Resource The AQL file that contains the error.
Path The path of the AQL file that contains the error.
Location The line number of the error.
Type The type of error that occurred.

The Problems view also provides AQL compiler warnings for slow regular expressions. These warnings identify complex regular expressions that cannot be run with the InfoSphere BigInsights Text Analytics accelerated regular expression evaluation engine (SimpleRegex) because the regular expression involves advanced constructs, such as look-ahead, look-behind, and boundary matches. Such regular expressions are evaluated with the generic Java engine that can be much slower.

In certain cases, it might be possible to simplify the regular expression to avoid the use of such advanced constructs altogether, or replace their usage with other AQL constructs with equivalent semantics. As a result, the simplified regular expression may become compatible with the accelerated engine resulting in higher extractor throughput. For details, see regular expression tips in Follow the basic AQL guidelines.

Outline view

The Outline view displays the outline of the AQL file that is open in the editor and lists the structural elements of the file. To add the Outline view to the current perspective, click Window > Show View > Other > General > Outline. In the Outline view, you can:
  • See a list of statements that are defined in the AQL file that is open in the editor.
  • Navigate to the statement definitions in the AQL file by clicking the statement.
  • Sort the statements alphabetically.

Hyperlink navigation

Hyperlink navigation is similar to navigating to the declaration of a Java™ element in a Java development environment. The AQL editor supports hyperlink navigation for the following identifiers:
  • Views, tables, dictionaries, and functions that are defined in the current file or module
  • Imported views, tables, dictionaries, and functions
  • Exported views, tables, or dictionaries and functions
With hyperlink navigation, you can navigate to the location where an AQL object is defined and open the specific dictionary or AQL file in an appropriate editor. The AQL editor also supports hyperlink navigation for names of external dictionary files and included AQL files.
For example, suppose that your AQL script contains the following statements:
					Line 1: import view MyView from module sampleModule;

Line 2: create view PhoneNumber as
Line 3: extract regex /\d+(\.\d+)?/  on R.text as match
Line 4: from Document R;

Line 5: output view sampleModule.MyView;

Line 6: export view PhoneNumber;
				

If you place the cursor anywhere inside the token PhoneNumber in Line 6 and click F3, the action takes you to the definition of the view PhoneNumber in Line 2. Similarly, if you place the cursor anywhere inside the token MyView in Line 1 or 5 and click F3, the action opens the AQL file that contains the definition of MyView and takes you to the line number in the sampleModule module where MyView is defined. Similarly, you can navigate to the definition of any AQL identifier by using F3 navigation across modules and projects.

When you attempt to navigate to the definition of a view whose source is not available or visible to the current build path of the project, the following message is displayed:
					There is no AQL source in project's build path that contains definition of this element.
				
Restriction:
  • If you edit a file and want to use hyperlink navigation for new or changed content, save the file first.
  • Hyperlink navigation is not supported for dictionary and AQL files that contain leading or trailing white space in the file path that is mentioned in the AQL code.

Content Assistance

The Content Assist feature provides context-sensitive assistance to complete the current AQL statement. For example, when you type the statement output view in the AQL editor and call the Content Assist feature, the tool displays all of the views that can be output at the current cursor location. You can then choose a view from the list to complete the AQL statement.

To see a list of all valid AQL statements for a specific location in the file:
  1. Place your cursor at the location in the file where you want assistance.
  2. Press Ctrl+Spacebar.
  3. Select the statement that you want added to the file from the list of valid AQL statements. The statement is completed automatically and added to the file. When Content Assist is started in a new blank line, a template of the selected AQL statement is added to the file.
  4. Modify the required information in the AQL statement or template that is inserted into your file.
Content assistance is provided for the following AQL constructs:
  • Export
  • Export dictionary
  • Export table
  • Export view
  • From
  • Include
  • Import
  • Import dictionary
  • Import function
  • Import module
  • Import view
  • Output view
  • Regex

Syntax highlighting

The AQL editor uses colors to highlight the different syntax elements in the AQL statements. The following table lists the colors that the AQL editor uses:
Table 2. Highlighting reference
Token type Color
Comments Maroon
Strings Green
Keywords Navy blue
Variable type Purple
Regular expression Pink
Default Black
For more information about the different token types in the AQL syntax, see AQL Syntax.

Regular Expression Generator

The Regular Expression Generator wizard helps you to create and refine a regular expression through a list of examples of the concept. Right-click a regular expression between the slash (/) characters in your AQL file, and select Regular Expression Generator to open the Regular Expression Generator wizard. In the wizard, you can generate or refine a regular expression. Click Finish, and the new expression replaces the old one. For more information, see Generating regular expressions.

Regular Expression Builder

The Regular Expression Builder helps you to build a regular expression by selecting constructs and combining them. Right-click a regular expression in your AQL file, and select Regular Expression Builder. The Regular Expression Builder opens with the regular expression as a parameter. Modify this regular expression by using the listed constructs. Click Finish, and the new expression replaces the old one. For more information, see Building regular expressions.

AQL Dependency and Reference Hierarchy

Similar to Java call hierarchy, the AQL editor can visualize the dependent relationships between AQL views that are defined in Text Analytics projects. These relationships are presented in the following views:
AQL Dependency Hierarchy
Shows a tree structure with a selected AQL view as the root and its children as other AQL views on which the root is dependent.
AQL Reference Hierarchy
Shows a tree structure with a selected AQL view as the root and its children as other AQL views that require the root in their definitions.

When you right-click the members in the tree, you can open the definition of that view in the AQL editor if the source of the view definition is available.

To start the AQL Dependency Hierarchy or Reference Hierarchy, select a view in the AQL editor, and select either Open AQL Dependency Hierarchy or Open AQL Reference Hierarchy. You can start the AQL Dependency Hierarchy or Reference Hierarchy only on views that are defined in the source directory of a project.