Advanced search¶

Date range search filter¶

The date range search allows users to search for any records whose active dates (e.g. dates of creation, accumulation, etc.) either overlap, or fall exactly within, a selected range. AtoM expects ISO 8601 formatted date values for searching - i.e. YYYY-MM-DD. Note that the fields include calendar widgets, to provide users with a graphical interface for date selection if desired - alternatively, you can manually enter your range directly into the text field. Whenever only YYYY values are entered, AtoM will automatically add -01-01 to values in the Start date field, and -12-31 to values in the End date field.

Important

When searching for a date range, AtoM searches against the values in internal, ISO-8601 startDate and endDate fields - those hidden from users, and formatted as YYYY-MM-DD, YYYY-MM, or YYYY. This is not to be confused with the Display date field, which is shown to public users, and allows you to use typographical marks to express approximation or uncertainty:

This means, if you have NOT included internal start and end date values, then your description(s) will not be returned when performing a date range search!

There are 2 ways of searching dates, represented by the radio button options in the Date range search interface - “Overlapping” (which is the default), and “Exact.”

The Overlapping option is a broadly inclusive search: any records whose dates overlap the user input range will be returned. For example, a search for a range from 1950-1970 would return descriptions with the following dates:

• 1945 - 1990
• 1945 - 1950
• 1970 - 1990
• 1955 - 1965
• 1955 - (no end date)
• (no start date) - 1980

The Exact option means that records must fall exactly within the specified parameters to be returned as a result. A search for 1950-1970 using the “Exact” option would not return any of the results listed above - but its results returned might include values such as:

• 1950 - 1955
• 1952 - 1969
• 1950 - 1970
• 1964 - 1969
• 1968 - 1970

To better clarify these options, some example diagrams have been included below to indicate the difference, illustrating how each option behaves for closed ranges (both start and end date included), and open ranges (where either a start or end date is included, and the second date is left blank). In the following images, results in green will be returned by the search query; results in red will be excluded from the results:

Closed range example¶

Overlapping:

Exact:

As you can see, the “Overlapping” option is very inclusive, while the “Exact” option is much more strict in terms of the results returned.

Open end date example¶

You can also input only a start date, or an end date, into the date range search if desired. If a user gives just a start date, it is interpreted to mean ‘filter to records that were active after this date’ - in other words, ‘end date of record > user supplied start date.’

Overlapping:

When the “Overlapping” option is used, records which start before the specified start date will be included if their date range extends into the specified parameter.

Exact:

When the “Exact” option is used, records starting before the specified start date range are excluded.

Open start date example¶

If the user gives just an end date, it means ‘filter to records that were active before this date’ - in other words, ‘start date of record < user supplied end date.’

Overlapping

When the “Overlapping” option is used for search with a specified end date but no start date, records whose ranges extend beyond the specified range may be included, if their start dates extend into the specified range.

Exact

When the “Exact” option is used, any record whose end date falls outside of the specified parameter will be excluded.

Using the date range search filter¶

When you place the cursor in the start date or end date fields of the date range search, a calendar drop-down menu will appear. This “datepicker” offers a graphical user interface for selecting the date, if desired.

Click on a day in the calendar to select that as your start or end date. You can also navigate through the months using the black forward and back buttons.

The month and year can also be adjusted via the two drop-down menus:

Alternatively, you can also ignore the calendar widget, and simply enter a date directly into the text field. AtoM expects the dates to be formatted as YYYY or YYYY-MM-DD. If you enter only the year, then AtoM will add -01-01 to start dates and -12-31 to end dates when the search query is submitted - for example, if you search for 1950 - 1970, AtoM will submit the query as 1950-01-01 (January 01, 1950) to 1970-12-31 (December 31, 1970). You can also specify an open range, by entering just a start date or just an end date.

Note

If you enter the dates in the wrong format, such as YYYY-MM, your search will return no results! In this case, AtoM will provide a warning above the date range fields:

Once you have entered your desired date range, use the radio button options to the right of the input fields to specify how the date range should be performed. “Exact” means that the start and end dates of descriptions returned must fall entirely within the date range entered, while “Overlapping” is much more inclusive, and any description whose start or end dates touch or overlap the target date range will be returned. See above for an illustration of the differences between Overlapping and Exact when returning results for:

• Closed range (start and end date)
• Open range (no end date)
• Open range (no start date)

If you click the (?) Help icon, a tooltip will appear below the fields with a brief explanation of the difference between “Overlapping” and “Exact.”

When you have entered all desired parameters, use the “Search” button to submit your query. Remember, you can also use other Advanced search filters or facets to further refine your search!

Back to top

Phrases¶

By default in AtoM, the Boolean search settings of AtoM are set to use AND as the default operator when multiple search terms are entered. This means that by default, a search for city hall will return results that include “city” AND “hall” - however, the order of terms doesn’t matter in the results returned, and they may not necesarily appear together.

To search for an exact phrase in AtoM, use double quotes to contain the terms you wish to search. For example, search “city hall” to return results that contain both “city” and “hall” together in that exact order.

If you wish to use “OR” as the operator between terms, see the section below, Boolean operators.

Result matches and field weighting¶

Each time a search is submitted in AtoM, the terms of the search are tokenized and results are ranked and ordered based on how relevant the match seems to the original search query.

To improve the relevancy of search results, AtoM includes some basic weighting added to certain fields, so that when a search term matches the data in one of these weighted fields, it is boosted and returned higher in the list of matching results.

For example, if you search for the word Toronto, then a record with “Toronto” in the title will be returned higher in the list of results than one that only has the word in a notes field. Below is a breakdown of how archival description fields are weighted in AtoM - the greater the weight, the more importance the match is considered and therefore the more it is boosted in the returned results:

• 10x weight: Title
• 6x weight: Creator
• 5x weight: Identifier, Subject access point, Scope and content
• 3x weight: Name access point, Place access point

Note that weighting in AtoM is cumulative, so a match in multiple weighted fields will increase amount the record is boosted.

For example, if you searched for city hall 123, this search would understood as: look for archival descriptions that contain city AND hall AND 123 in any field. If I have a description that has an identifier of “123,” the word “city” in the title, and the word “hall” in the scope and content field, this record would be given a weight of +20 - that is, +10 for matching the word “city” in the title; +5 for matching 123 in the identifier, and +5 for matching “hall” in the scope and content field. However a record with “city hall” in the title and 123 in the Scope and content might be returned higher, as it would receive +10 for matching “city” in the title, another +10 for matching “hall” in the title, and +5 for matching 123 in the scope and content field.

Note that there are many complex factors that go into how search relevance is determined and ranked in Elasticsearch, and it may not always be immediately apparent why some records appear higher in the returned results than others. For example, the number of other words in a given field will affect how elasticsearch ranks the relevance. So a search for “photographs” might return a result higher in the list that ONLY has the word photographs in the scope and content, than one where the word “photograph” appears in a 20-word long title - even though the added weights to the title field are technically higher than those added to the scope and content field.

Boolean operators¶

In AtoM, Boolean search operators are supported in the search box and in the Adanced search menu. Boolean searching is a particular application of what is known as Boolean logic, a subset of algebra used for creating true/false statements. Since computers operate in binary (using ones and zeroes), computer logic can often be expressed in boolen terms (true/false). Boolean expressions use a number of operators, the most common of which are AND, OR, and AND NOT. Using Boolean operators in terms of search queries (i.e. Boolean search) allows a user to limit, widen, or otherwise define a search in granular terms - for example, searching “fonds OR collection” would widen a search to include results that have either term.

Using the Boolean operators available in AtoM allows users to build complex queries from anywhere in AtoM using the general search box located in the AtoM header bar, without having to navigate to the Advanced search interface.

Using Boolean operators in the AtoM search box:

• Use AND to combine search terms to narrow your search: e.g. search vancouver AND "city hall" to return only descriptions that contain both search terms
• Use OR to to combine search terms to broaden your search: e.g. search vancouver OR "city hall" to return descriptions that contain either search term
• Use AND NOT to exclude terms: e.g. search vancouver AND NOT "city hall" to return descriptions that include vancouver but do not contain the phrase “city hall”

Other Boolean operators available in AtoM:

For further examples of the use of these Boolean operators, users can consult the Zend Lucene search documentation as Elasticsearch is built on the same Apache Lucene base as the Zend framework. Developers interested in fine-tuning these settings, or technical users interested in how Elasticsearch operates may wish to consult the query string query Elasticsearch reference documentation for more information on Elasticsearch’s default behaviors, and possible configurations.

Introduction and basic usage¶

AtoM currently uses Elasticsearch version 1.7 - you can find the documentation for AtoM’s ES version here:

• https://www.elastic.co/guide/en/elasticsearch/reference/1.7/index.html

The index is organized hierarchically so that related elements are nested within broader categories. These are expressed in the ES field names as periods separating the name elements. For example, AtoM’s alternative identifiers consist of 2 fields - a label and an identifier value. In the ES index, these are expressed as:

alternativeIdentifiers.identifier
alternativeIdentifiers.label

To use the Elasticsearch field names below as part of a query in the AtoM user interface, use the following basic syntax:

field.name:searchterm OR field.name:"search term"

Use quotes for search multiple terms in a specific field; quotes not necessary when searching for a single term. For example, to search for an alternative identifier value of 123, you could enter the following search term into the AtoM search box:

alternativeIdentifiers.identifier:123

But to search for an alternative identifier label called “Legacy call number,” enter:

alternativeIdentifiers.label:"Legacy call number"

Because AtoM is a multilingual application, some translatable fields are organized together under the common prefix of “i18n” (a common abbreviation in multilingual application development, short for internationalization). When searching these fields, the culture of your search must be entered as part of the field name, because nested within each i18n element of the index there will be a different term for each language.

In the full field list below, some fields listed include %LANG% - this is a placeholder indicating that you must insert the specifc 2-letter ISO language code for the culture you want to search in - e.g. en for English, fr for French, etc. For example, the Extent and medium field is listed below as: i18n.%LANG%.extentAndMedium.

Therefore, to search your English data for “linear feet” in the Extent and medium field, you would enter the search term as follows:

i18n.en.extentAndMedium:"linear feet"

Searching for empty fields¶

You can perform a field-specific search for fields that do not have any data added to them (i.e. which are empty) using _missing_. The basic structure of the syntax is like this:

_missing_:field.name

For example, to search for English archival descriptions with no data in the scope and content, use the following as your query: field:

_missing_:i18n.en.scopeAndContent

Searching for populated fields¶

Similarly, you can also perform a search that will return all records that include data in a specific Elasticsearch field, without having to provide a specific keyword as a parameter. The following syntax structure can be used to return results that have data in the specified field:

_exists_:field.name

For example: to search for all English archival descriptions that have data in the “Finding aids” field:

_exists_:i18n.en.findingAids

Searching for recently created or modified descriptions¶

First, remember that the “Most recent” option of the Sort button found on the search/browse page will change the sort order of displayed results so that those most recently created or modified are shown first - for more information, see: Sort button. Additionally, an administrator can use the Description updates module to search for either created or modified (or both) records withing a specific date range - for more information, see: Description Updates.

However, there is a way to query the ES fields directly - The 2 Elasticsearch fields are called createdAt and updatedAt. When a new record is created, createdAt and updatedAt are initially the same. After subsequent edits, the createdAt will never change, while the updatedAt value is updated to the most recent time. Here are some example queries, which can be used in the AtoM global search box:

updatedAt:[2010-01-01 TO 2012-12-31]

This will return all records last modified between Jan 1 2010 and Dec 31 2012

updatedAt:[* TO 2012-01-01]

In this case, the asterix is a wildcard. This will return all records modified before 2012-01-01. For more information on wildcards and other special characters for Boolean searching, see above, Boolean operators.

createdAt:[2012-01-01 TO *]

This will return all records created after January 1, 2012.

Searching for digital objects¶

The digital object fields in the Elasticsearch index are:

digitalObject.filename
digitalObject.mediaTypeId
hasDigitalObject
digitalObject.thumbnailPath
transcript

hasDigitalObject is a binary field - 1 = yes, 0 = no. For example:

hasDigitalObject:1

Will return results that have a digital object attached.

the mediaTypeId is an internal ID number for different types of digital objects. The default values used in AtoM for each media type are:

• Audio: 135
• Image: 136
• Text: 137
• Video: 138
• Other: 139

For example, to return results with a video linked:

digitalObject.mediaTypeId:138

The transcript is the indexed text captured from the text or OCR layer of a PDF or other text-based document uploaded as a digital object. Note that the Advanced search panel has an option to limit a search to transcript text. For more information, see above, The Advanced search panel.

Publication status¶

AtoM includes a publication status for archival descriptions, that allows users with sufficient permissions to hide “draft” records from public view, or mark descriptions as “published” so they are publicly visible to logged out users. For more information, see: Publish an archival description. Administrators can also set a global default status for new records via Admin > Settings - for more information, see: Default publication status. Finally, an administrator can also use the Description updates module to identify Draft descriptions - for more information, see: Description Updates.

However, you can also use the related Elasticsearch field to search for all published or draft records directly. Note that to return results marked “draft,” you must be authenticated (i.e. logged in) with permissions to view drafts. (see Edit permissions for more information).

Draft and Published are terms in a locked taxonomy in AtoM, and you can search for them by using their internal ID values. In AtoM, 159 is the default term ID for Draft, while 160 is the default term ID for Published. We can use AtoM’s publicationStatusId Elasticsearch field to perform a search for all draft records, or all published records:

To perform a search for all draft archival descriptions:

publicationStatusId:159

To perform a search for all published archival descriptions:

publicationStatusId:160

List of Archival description ES fields¶

Not all fields available in the archival description edit templates are added to AtoM’s search index. Additionally, some fields are queried not via Elasticsearch, but by searching directly against the database or via another method in the AtoM code. Consequently, the list below is not exhaustive, but it should cover the majority of available fields. You can use the syntax described in the introductory section above to construct a query against a specific field.

alternativeIdentifiers.identifier
alternativeIdentifiers.label
createdAt
digitalObject.filename
digitalObject.mediaTypeId
digitalObject.thumbnailPath
findingAid.status
findingAid.transcript
hasDigitalObject
i18n.%LANG%.accessConditions
i18n.%LANG%.accruals
i18n.%LANG%.acquisition
i18n.%LANG%.alternateTitle
i18n.%LANG%.appraisal
i18n.%LANG%.archivalHistory
i18n.%LANG%.arrangement
i18n.%LANG%.edition
i18n.%LANG%.extentAndMedium
i18n.%LANG%.findingAids
i18n.%LANG%.institutionResponsibleIdentifier
i18n.%LANG%.locationOfCopies
i18n.%LANG%.locationOfOriginals
i18n.%LANG%.physicalCharacteristics
i18n.%LANG%.relatedUnitsOfDescription
i18n.%LANG%.reproductionConditions
i18n.%LANG%.revisionHistory
i18n.%LANG%.rules
i18n.%LANG%.scopeAndContent
i18n.%LANG%.sources
i18n.%LANG%.title
identifier
levelOfDescriptionId
materialTypeId
publicationStatusId
referenceCode
referenceCodeWithoutCountryAndRepo
slug
sourceCulture
transcript
updatedAt

generalNotes.i18n.%LANG%.content
alphaNumericNotes.i18n.%LANG%.content
conservationNotes.i18n.%LANG%.content
physicalDescriptionNotes.i18n.%LANG%.content
continuationOfTitleNotes.i18n.%LANG%.content

Additionally, there are some related fields belonging to linked entities that, when linked to an archival description, can be searched to return related archival descriptions. These include repository records, authority records (linked as either a creator or name access point), and subject, place, and genre access points.

For example, to return all English archival descriptions where “Jane Doe” is linked as the creator, you could enter:

creators.i18n.en.authorizedFormOfName:"Jane Doe"

Here is the list of related entity fields that are searchable in the global search box, that will return linked archival description results:

creators.i18n.%LANG%.authorizedFormOfName
creators.otherNames.i18n.%LANG%.name
creators.parallelNames.i18n.%LANG%.name
creators.standardizedNames.i18n.%LANG%.name
genres.i18n.%LANG%.name
names.i18n.%LANG%.authorizedFormOfName
names.otherNames.i18n.%LANG%.name
names.parallelNames.i18n.%LANG%.name
names.standardizedNames.i18n.%LANG%.name
places.i18n.%LANG%.name
repository.i18n.%LANG%.authorizedFormOfName
subjects.i18n.%LANG%.name

Back to top