CITES | University of Illinois

Lens API Quick Start Guide

For IT Pros
This page contains information about Lens, a campus network information system.

Required access credentials

Access to the Lens API requires a current campus Active Directory user account whose name is known as a network contact in CITES Contacts Database. The Lens API provides users with data pertaining to the networks for which they are listed contacts.

Info icon Not just for Net IDs anymore!

We have implemented an authentication mechanism that recognizes user accounts anywhere in the Campus Active Directory. Accordingly, the Lens API can service any Campus AD user that is also known to Contacts Database as a network contact. This allows you to grant privileges to pseudousers in your OU for use by departmental scripts and applications. See below for details.

 

Access by Net ID

Your Net ID is represented by an [AD.UIUC.EDU] Active Directory user account which identifies you personally. As a network contact, you can access the Lens API using your Net ID and AD password. This is appropriate for your direct, interactive use of the API because you can supply your name and password as necessary. For example, you can use your Net ID with command-line utilities you create and for direct browser-based access.

Other applications require Lens API access when no user is involved. For these we do not recommend using your personal Net ID and AD password when such use requires you to place them in persistent storage (e.g. a disk file).

Storing your password on disk risks more than unauthorized Lens API access. At potential risk are any other accessible University resources with which you are entrusted.

See the following section for using the Lens API with unattended applications.

Access by pseudouser

Some applications use the Lens API when no user is involved who can provide access credentials (e.g. scheduled jobs). For these you can use a campus AD user account you create within your own organizational unit and dedicate to this purpose. This pseudouser account represents your application and neither you nor any other individual person. You can store the pseudouser password for your unattended application without risk to personal and unrelated resources.

We strongly recommend you choose a username that begins with your unit
or group followed by the hyphen character
in order to identify the
responsible unit and sidestep potential conflicts with Net IDs. (For
several years, new Net IDs have been created to exclude the hyphen
character.) For example, pseudousers for the Physics or Computer Science
departments might be named "physics-lens-user" or "compsci-lens-user".

Info icon Where in the Active Directory is Carmen Sandiego?

Some units operate private Active Directory trees which may or may not be somehow linked to the UIUC campus AD. Although these trees may "trust" the Campus AD and thereby recognize its users and objects, the reverse is not generally true. You need to define your pseudouser account in the UIUC Active Directory. Your unit may already own space in the UIUC AD. Contact ADSupport@illinois.edu for assistance with the UIUC AD.

As for any Net ID, the Lens API imparts privileges to your pseudouser account matching its CITES Contacts Database network contact assignments. Unlike the Net IDs, however, Contacts Database does not automatically know that your pseudouser exists. You need to contact net-trouble@illinois.edu to request a Contacts Database identity made with the same name as your non-Net ID pseudouser account. Afterward, you or an authorized network contact in your unit can assign your pseudouser to be read-only contacts for desired networks.

We strongly recommend you assign read-only privileges! Although today the Lens API just provides read-only service, the pseudouser is also known to and carries privileges within Iris! Furthermore, the Lens API may not always remain read-only.

Through pseudouser accounts, you can automate your applications with little need for contact beyond your unit. Although they mitigate risk, they nonetheless access University resources and can be exploited to compromise privacy. When you create and empower pseudouser accounts as agents on your behalf or your unit's, you retain responsibility for its proper use and protection. In short: you made it, you own it, you authorized it, and you are responsible for it.

General query format

The Lens API defines several types of objects which may be returned by HTTP GET queries. (The POST method is not supported at this time. See the Lens API FAQ.)

All objects of like type provide the same set of attributes. For example, device objects all have a device_name attribute. These attributes may be used in queries to retrieve objects, and they are returned by queries as a part of the objects. Furthermore, objects are related to each other. For example, device objects are related to interface objects (because network devices have interfaces). The goal of every Lens API query is to fetch a list of some particular type of object based on the values of attributes and their interrelationships. The list may be "dressed up" with additional information to build a consistent and informative data set in one query.

All Lens API queries have a format described by the following template:

https://lens-api.cites.illinois.edu/lens/uiuc-lens/<subject type>?
	 <param1>=<expression1>&<param2>=<expression2>&...&dressings=<dressing1>,<dressing2>,...

The subject type is the type of the objects you want to list (e.g. device). Each parameter can be the name of an attribute that is found in the subject type (e.g. param1 may be device_name). An expression describes the attribute values you're looking to match in choosing objects for your list. Alternatively, a parameter can describe how the subjects in your list must relate to other objects. The list of objects reflects the application of all these qualifications as described below. Finally, the result is elaborated with additional data as requested by the given set of "dressing" tokens (e.g. dressing1).

Example: device

To retrieve Lens device object data, point your browser to

https://lens-api.cites.illinois.edu/lens/uiuc-lens/device 

You will be prompted for your netID and AD password in ad-dc-t1.ad the first time you submit a request. The URL above retrieves all devices stored in the Lens database that include vlans of your privileged network. The device attributes returned will be the "essential" attributes listed in the device specification.

Supported object types

Here is a list of all currently supported object types, linking to their specifications. Consult each specification to understand what the type represents, the attributes it supports, and how its objects relate to others.

Query mechanics

Object attribute data types

Objects such as devices can also be retrieved using any of their attributes as search criteria. The attribute names are listed in the specification, as are their basic data types. An attribute's data type defines the formats for its query expressions and its values as returned by queries. The data types currently supported are:

Datatype Value Range Note
Text 256 character string treated as a case-insensitive exact match. If the string begins with the tilde "~" character, the text is treated as a regular expression
Integer
Decimal
38 significant digit number, or "MIN" or "MAX" token "MIN" and "MAX" represent negative and positive infinity
Timestamp
Date
UTC (GMT) time, or "MIN" or "MAX" token All dates in and out of webservice are in UTC. Before you query, you must translate local times to UTC in order to get the results you expect. Likewise, you must translate time values returned by queries back to local time.  Symbols "MIN" and "MAX" represent the earliest past and farthest future times recognized (1/1/1970 and 1/2/2037)
Boolean 1 or 0 Boolean values are represented as the integer 1 for TRUE and the integer 0 for FALSE
IP address dotted-decimal IP address or subnet in CIDR notation If a single IP is given, an exact match will be performed. If a CIDR notation is given (e.g. 192.168.100.0/24), it will perform a fully inclusive range match. Note also that a CIDR expression will be converted into the appropriate "power-of-2" subnet for the range scan, so both "192.168.100.55/24" and "192.168.100.0/24" will result in the same scan of the range 192.168.100.0 through 192.168.100.255.
MAC address 12 hexadecimal characters Any delimiter is allowed, although data is always returned without any delimiter. MAC fields ONLY perform exact matches.

Regular Expressions

The webservices supports POSIX Extended Regular Expressions for all text data types. To invoke a regular expression match, the string must begin with the tilde "~" character. The API forces all regular expression matching to be case-insensitive. The POSIX Extended Regular Expressions or ERE flavor is a standard similar to the one used by the UNIX egrep command. For a more complete description of POSIX ERE regular expressions, see POSIX Extended Regular Expressions. For a table comparing various regular expression flavors, see Regular-Expressions.info.

Range queries

The webservices recognizes the comma "," as a range separator for all numeric, date, or scalar IP data types. If a comma is used, the webservice will perform a fully-inclusive range match. For example, using "vlan_numer=100" will result in an exact match for vlan_number 100, while using "vlan_number=100,200" will result in a range scan for all vlan_numbers between 100 and 200, inclusive. IP datatypes allow single IP range scans, such as "192.168.100.0,192.180.100.255", but do not allow CIDR-notation mixed with the comma range separator ("192.168.100.0/24,192.169.101.255") as the desired outcome is unclear.

Semantics of joining multiple parameter queries

In all cases, different parameters will be "AND"ed, while multiples of the same parameter will be "OR"d. So "device_name=sw-dcl1&type=switch" translates to "device_name=sw-dcl1 AND type=switch" while "device_name=sw-dcl1&type=switch&device_name=sw-dcl5" translates to "(device_name=sw-dcl1 OR device_name=sw-dcl5) AND type=switch".

Relationship-based queries

All object types have relationships defined in their specification. Each relationship has a name and represents a real-world semantic link between an object and others. For example, a device object relates to several interface objects via its interfaces relationship. A relationship has only one "related type". Objects can be queried using relationships and the attributes of related types by the following syntax:

<{{subject_type}}>?<{{relationship_name}}>:<{{related_object_attribute}}>=<expression> 

A query by relationship lists the objects (of subject_type) which relate (via relationship relationship_name) to a selection of objects (of the related type) matching the attribute query expression. Note that you may build your queries with simultaneous qualifications by attributes and by relationship, subject to the multiple-parameter semantics described in the previous section.

For example, to find all devices the have an interface that is named "A1", you would use the following query:

device?interfaces:ifname=A1 

device is the base object, interfaces is a device relationship that references interface objects, and ifname is an attribute of an interface.

Note that relationships function similarily to attributes in the syntax above, making it possible to follow a "chain" of relationships to a distantly related object. For example, one could extend the previous example to find all devices that have an interface that is configured for vlan 32 untagged by using this query:

device?untagged_interfaces:vlans:vlan_number=32

The device object's "untagged_interfaces" relationship points to interface objects, which themselves have a relationship called "vlans" which points to interface_vlan objects that have the attribute "vlan_number." This query will return all device objects that include in their set of "untagged_interfaces" an interface that has within its set of "vlans" an interface_vlan object with vlan_number=32.

Object dressings

A basic query will only return a list of objects of the subject type that satisfies the query parameters. In order to get related objects returned in the same query, you must apply "dressings" to the result. Each Relationship defined for an object in the specification provides a "Dressing Parameter" to invoke the dressing. Dressings are requested by including the "dressings" parameter followed by a comma-separated list of dressings in the order you wish them applied to your result. For example, to request devices with device_name=sw-dcl5 and all of that device's interfaces:

device?device_name=sw-dcl5&dressings=dev_interfaces 

The objects returned in a dressing call can also be dressed. To request the same device with all its interfaces and those interfaces' vlans:

device?device_name=sw-dcl5&dressings=dev_interfaces,ifc_vlans 

Note that the ordering of the dressings listed in the request is important. When the "dev_interfaces" dressing is applied, interface objects are added to the result set. The "ifc_vlans" dressing is then applied to those interface objects. If "ifc_vlans" is requested first, there would be no interface objects in the result set to modify, so that dressing would be empty. Dressings are always applied to every object of the dressing's target type in the result set, no matter where in the list that object was applied.

Info icon The difference between Query-by-Relationship and Dressing

Objects' relationships provide the bases both for query-by-relationship and for dressing results. However, these are distinct operations which do not otherwise interact. Query-by-relationship does not automatically dress the list of subjects with the matching related objects. For example, if you build a list of devices that carry a particular VLAN, you still have to request the list be appropriately dressed if you want the VLAN information returned with the devices. Likewise, the dressing does not reflect the query-by-relationship. If you ask for VLAN information with the devices, you get information for all VLANs per device. (However, your desired result is in there.) In other words, query qualifications simply function to build the subject list and do not imply nor interact with potential dressings.

 

How the API returns data

Result structure

See the following link to learn about the data structures returned by the Lens API. Note: This page is taken from early design stages. While the result structure is accurately defined, please note that the examples do not reflect the true names of object types and attributes. You'll still get the idea.

Lens API - Result Object

Serialized data encodings

In order to package an interrelated and potentially large data set, the Lens API returns a complex data structure. The data structure is serialized in one of the available encodings:

Encoding MIME type Description
XML text/xml The result is encoded in an XML document with a very simple nested-element format. It may be trivial to decode into native structures with a basic SAX parser implementation. Note: There is no DTD nor XML Schema definition available.
JSON application/json The result is encoded in a JSON program which can be deserialized into native structures using a Javascript interpreter or dedicated JSON parser.
YAML text/x-yaml The result is encoded in a human-readable language-independent YAML document which can be deserialized into native structures with a standard YAML parser. NOTE: YAML is not as widely known as XML and JSON. Given the availability of the latter, YAML is not considered a supported technology. However, the API developers intend to preserve the YAML interface for some technical advantages and its human usability.

Selecting an encoding

You can select an encoding by the following means:

  • Include a Content-Type header in your HTTP request with a value indicating the MIME type of the desired encoding. For example:
Content-Type: application/json
device?device_name=~sw.*&content-type=text/xml 
  • Supply a content-type URL query parameter with a value indicating the MIME type of the desired encoding. For example:

More examples

The root url for all examples is: https://lens-api.cites.illinois.edu/lens/uiuc-lens/

device?type=switch&opsys=hp&opsys=os 

The above query returns devices whose type="switch" and (opsys="hp" or opsys="os").

device?type=~^Swi&opsys=~^hp&opsys=os 

The above query returns devices whose type satisfies the regular expression /^Swi/ and (opsys satisfies /^hp/ or opsys="os"), all case insensitive. Regular expressions can be used for string columns such as name, type, created_by, model, opsys, by prefixing the string with the tilde "~".

device?ip=172.20.14.47 

The above query returns devices with an IP Address of 172.20.14.47

device?ip=172.20.14.0/23 

The above query returns devices with an IP Address in the range of 172.20.14.0 through 172.20.15.255

mac_port?mac=001122334455&mac=aabb1122ccff 

The above query returns a list of current ip_mac observations where the MAC address matches either 001122334455 or aabb1122ccff

ip_mac?ip=172.20.14.47&ip=172.20.14.48 

The above query returns a list of ip_mac objects whose IPs match either 172.20.14.47 or 172.20.14.48.

Scripting

In a scripting language, such as perl, you can specify the return mime type as well as your credentials to allow httpd basic authentication over SSL.

#!/usr/bin/perl -w
use strict;
my $url='https://lens-api.cites.illinois.edu/lens/uiuc-lens/device?device_name=sw-dcl5';

my @results = get($url);
foreach my $result (@results) {
  print $result."\n";
}

sub get {
  use LWP::UserAgent;
  my $addr = shift;
  my $req = HTTP::Request->new(GET => $addr, HTTP::Headers->new);
  $req->header(Accept => 'text/xml');  # substitute your favorite mime type here
  my $auth = "Basic ".ncode("mynetid:mysupersecretpassword");  # netID and ad password
  $req->header(Authorization => $auth);
  my $res = LWP::UserAgent->new->request($req);
  die $res->status_line unless $res->is_success;
  return $res->content;
}
sub ncode {
  use MIME::Base64;
  my $in = shift;
  return encode_base64($in);
}
1;

The result displays in the standard output. Currently, text/xml, text/x-data-dumper, text/x-json, text/x-yaml mime types are supported.

 

 

Last updated Thursday, July 18, 2013, 5:14 pm