The Civil War Governors of Kentucky Digital Documentary Edition was initially attracted to Omeka as a temporary publication platform that would deliver document facsimiles, transcriptions, and limited metadata to meet the demand of interested researchers while a more robust interface that could link the documents together using networks of identified and annotated people, places, organizations, and geographical features could be developed. The pre-existing set of tools to achieve these goals and the active development community made Omeka a cost- and time-effective intermediate publication solution.

CWGK manages its editorial work through DocTracker, a FileMaker based editorial application that exports records in TEI-XML. TEI-XML documents consistently record and represent a text and metadata about the text's form, history, and provenance. Regardless of the publication platform chosen, TEI-XML would be the basic output format for CWGK.

CWGK images the documents the project accessions at 300 dpi TIFF files, but chose to group images and convert them to single-document multi-image PDF files for day-to-day editorial work. To save an extra step of converting these PDFs into another format for web display, CWGK hoped to use these PDFs in its web interface.

CWGK and developer/designer Anneliese Dehner wanted to ensure that identified front-end users (students, teachers, researchers, and genealogists) could locate and read a document in the following ways:

• Simple keyword searches
• Faceted navigation
• Browsing within a repository
• Browsing within a collection
• Advanced searches in any combination of the following fields: accession number, collection, date of creation, dates mentioned, document genre, document title, editorial note, item location, place of creation, repository, transcription
• Browsing exhibits which feature documents
• Browsing featured collections from the home page

And CWGK wanted the following back-end features for its own staff:

• Upload new XML documents as editorial work is completed
• Monitor site usage
• Create new collections
• Edit transcription metadata and mark-up
• Create exhibits
• Feature collections and exhibits on the home page

All of which Omeka could do with its core set of tools and capabilities or with an arrangement of plugin and third-party tools.

Collection Tree

Rationale: In DocTracker, Civil War Governors of Kentucky simulates a hierarchical organizational structure (Source / Collection / Document). When migrating to Omeka, Early Access required a similar hierarchical structure in its back-end architecture and its front-end navigation. By default, Omeka offers two levels of hierarchy (Collection / Item). And the organizational structure of Omeka collections is flat, by default. Installing and activating the Collection Tree plugin meets the Early Access requirement by adding  additional levels to Omeka’s hierarchical structure. After configuring the Collection Tree plugin, the Early Access structure looks like this: Parent Collection / Child Collection / Item. The parent collection is a CWGK repository, equivalent to “Source” in DocTracker. And the child collection is a CWGK collection, equivalent to “Collection” in DocTracker. The Collection Tree is visible in front-end repository pages, like this one: http://discovery.civilwargovernors.org/collection/KYR-0001, as an alphabetical list of collections housed in the repository.

Configuration: The default Collection Tree plugin output is an un-ordered list of nested collections. The plugin has been modified to instead output divs for each child collection on the parent collection (repository) page. By default the “tree” displays on every front-end collection page, but it has been hidden on child collection pages.

CSV Import

Rationale: Early Access required migrating ~10,000 TEI documents from DocTracker to Omeka and importing ~10,000 pdf files from Kentucky Historical Society shared drives to Omeka. Files are to be migrated/imported in batches. Each TEI document and its respective PDF file should map to the same Omeka item. Installing and activating this fork of the CSV Import plugin meets these requirements as it adds the options to upload pdf files in batches, to map each pdf file in the batch to an existing Omeka item, and to choose the match point between pdf file and Omeka item. The CSV Import plugin follows the Dropbox TEI plugin (below) in the migration workflow from DocTracker to Omeka (see 5.0 User Guide for more information about the migration process)

Dropbox TEI

Rationale: In DocTracker, CWGK editors transcribed tens of thousands of documents and encoded them for export as TEI documents (XML). Early Access required migrating ~10,000 of these documents from DocTracker to Omeka. Files are to be migrated/imported in batches, and metadata from the TEI header should map to Dublin Core metadata fields in Omeka. Once populated, Omeka metadata fields become access points for CWGK documents. And these access points enable researchers to discover documents of interest. In addition to data mapping, Early Access required that the marked-up transcription (the TEI text element) displays as styled HTML. In order to meet these requirements, Dehner investigated several potential solutions:

• The Bookmeka plugin has potential. It offers several formats of the TEI transcription, and its README.md file promises CSV Import integration. It currently exists as a beta option, but could be explored further in the future.
• In 2012 the TEI Display plugin was updated to be compatible with Omeka 2.x, but it was never publicly released. Dehner hoped that its beta status would be sufficient for the needs of Early Access, but the plugin did not correctly map metadata from the TEI header to Omeka and the plugin does not support batch imports.
• The XML Import plugin requires converting XML files to CSV files before batch importing. And because CWGK sought a more streamlined process, Dehner eliminated this option.
• TEI Boilerplate does an excellent job transforming TEI to HTML, and Dehner adapted its functionality for Early Access (see XSLT below)
• Sue Hemmens, of Dublin Ireland’s Marsh Library, developed a plugin that combines aspects of the TEI Display and Dropbox plugins, but it is only compatible with Omeka 1.x, an older and vastly different version of Omeka. Her unreleased plugin batch uploads TEI files, creating Omeka items and mapping the TEI header to Omeka metadata fields upon import. She graciously shared her notes with me, and Dehner rewrote the plugin to be Omeka 2.x compatible and to map fields identified by the CWGK team (see Data below). Dehner also added functionality to render the TEI transcription as HTML (see XSLT below). This solution meets the requirements of Early Access. (see User Guide below for the workflow process)

This plugin was written by Anneliese Dehner. It is a mash-up of Omeka’s Dropbox and TEI Display plugins. The Dropbox plugin brings batch upload functionality while the TEI Display plugin brings XML mapping and also renders the TEI transcription as HTML.

Clean Url

Rationale: By default, Omeka identifiers and terms (items, collections, etc.) are combined to create a url template. For example, the default url for an Omeka item is: discovery.civilwargovernors.org/items/show/72 and the default url for an Omeka collection is discovery.civilwargovernors.org/collections/show/6. Early Access required a url structure that includes the CWGK accession number. Installing and activating the Clean Url plugin meets this requirement. After configuring the Clean Url plugin, the front-end url template for a CWGK Omeka item is discovery.civilwargovernors.org/document/KYR-0001-001-0001 and the front-end url template for an Omeka collection is discovery.civilwargovernors.org/collection/KYR-0001-001. The back-end url structure, however, follows the default Omeka template.

Default Sort

Rationale: Early Access required that documents be sorted by CWGK accession number (eg. KYR-0001-001-0001), in ascending order. By default Omeka sorts items by the date they were added to Omeka, in descending order. While it is possible to configure sorting options (sort by accession number, title, date, etc.) in some front-end Omeka page templates; it is only possible to change the default sort field by editing Omeka’s core files. Editing Omeka’s core files is not recommended because it complicates the process of upgrading to new versions of Omeka. Installing, activating, and configuring the Default Sort plugin meets this requirement. In front-end pages, items are sorted by ascending CWGK accession number.

Solr Search

Rationale: Early Access required faceted navigation. Omeka’s default search includes pre-filtered searching in the form of advanced search, but it does not include faceted navigation. Solr Search is a popular and powerful search platform, offering endless configuration possibilities. Installing, activating, and configuring the Solr Search plugin meets the faceted search requirement and gives the CWGK staff the flexibility to improve search results in the future if they choose to investigate Solr’s configuration options. Activating the Solr Search plugin replaces Omeka’s simple search box with Solr’s search box. This action also replaces Omeka’s default search results page with Solr’s results page, which includes a left sidebar of facet options. The Solr Search plugin does not offer an advanced search. This has been added back in separately (see Data & Directories below) and Omeka’s Advanced Search has been disabled.

PDF Embed

Rationale: Early Access requires presenting the scanned document beside the document’s transcription. The document viewer should be clean and minimal so the viewer’s frame does not interfere with the user’s interaction with the document. The viewer should allow the user to zoom in and out, to rotate the document, and to pop-out the document in another window. The viewer should also protect the security of the document, not allowing the user to download or print an original copy. All of this should work smoothly across devices and browsers. Dehner explored several possibilities to meet this requirement:

• Dehner first installed the Docs Viewer plugin because of its responsive, minimal appearance. However this viewer allows the user to open the pdf in Google Drive, making the original pdf available for download or printing. Because the entire viewer is packaged as a Google product, it is not possible to limit these functions. Due to security issues, this viewer was abandoned.
• The OpenSeadragon plugin has potential for image files, but it is not compatible with multi-page pdfs.
• While the Universal Viewer plugin offers many features, the viewer frame is overloaded with buttons. Dehner eliminated this viewer due to its bulk, which detracts from the document itself.
• The PDF Embed plugin is based on pdf.js and is simple in appearance but feature-rich. Its configuration is flexible; features can be disabled and customized without slowing the viewer’s functionality. Dehner chose this plugin, as it meets the project requirements. The plugin can be configured to allow user download and printing. But these features have been disabled in Early Access to meet the document security requirement.

Simple Pages

Rationale: The CWGK staff writes historical and editorial information that supports and complements the collection. Early Access should include web pages that house this information. The Simple Pages plugin meets this requirement. The plugin provides a WYSIWG text editor to easily create rich text web pages, and it allows staff users to simply add these pages to the navigation menu. The WYSIWG text editor can also be disabled so staff users can build pages with HTML, if they prefer.

Exhibit Builder

Rationale: CWGK is interested in considering Omeka as a platform for future curated exhibits and lesson plans drawn from its documents. The Exhibit Builder plugin meets this requirement as it offers endless template options for laying out text, images, and Omeka items.

Simple Contact Form

Rationale: Because Early Access publishes transcriptions which are not yet considered authoritative, the editors invite users to suggest transcription corrections. This requirement is met by the installation and activation of the Simple Contact Form plugin, which creates a contact form on a page of the Omeka website. A link to the contact form is located below each document transcription. The form forwards each form submission to a staff email account. Recaptcha has been activated in Omeka to prevent spam submissions via the form. 

XSLT

Rationale: Early Access required that TEI document transcriptions display as HTML. XSLT is required to transform XML to HTML. The TEI Boilerplate simplifies the rendering process by handling many basic transformations. The boilerplate XSLT is built out to transform special CWGK cases like stamps, columns, and marginalia. The stylesheet is called by the Dropbox TEI plugin (above) to render the transcription of the XML file uploaded to the Omeka item via Omeka’s dashboard.

TCPDF

Rationale: Early Access required making documents available for public viewing. The documents can be downloaded if they are kept within the context of the collection. This is achieved by adding the KHS logo, the CWGK logo, and a right to use statement to the pdf file at the point of download. TCPDF is a PHP class for generating PDF documents. This is one part of the solution to create branded PDFs on the fly. The process also requires TCPDI, TCPDI_PARSER, and FPDF_TPL (all below)

TCPDI

Rationale: Because TCPDF does not import existing PDF documents, an importer is required. CWGK PDF documents are version 1.6. And TCPDI can import existing PDF documents up to version 1.7. However TCPDI requires TCPDI_PARSER and FPDF_TPL to import PDF documents.

TCPDI_PARSER

Rationale: The parser is required by TCPDI to import existing PDF documents.

FPDF_TPL

Rationale: This class is required by TCPDI to read pages from existing PDF documents.

Google Analytics

Rationale: CWGK is interested in capturing how users interact with the site. Do users enter the site from Facebook, from the CWGK WordPress site, from Google searches, etc.? Where do users go after viewing an Omeka exhibit? Do they browse or search? Do they start with an advanced search or a simple keyword search? Do they download xml files? Do they download pdf files? These are some of the questions CWGK staff hope to answer with site metrics. Adding Google Analytics tracking code to the footer of Early Access meets this requirement as it gathers data about site visitors and their behavior as they navigate the website.

Overall, Civil War Governors of Kentucky is very pleased with Omeka as a publication platform. In fact, the project plans to continue to expand the size and functionality of the Omeka site rather than develop a new beta site to handle annotated documents.

Themes and Design

Developer/designer Anneliese Dehner created a custom Civil War Governors theme for the site. Omeka themes, like those in WordPress or other common sites, arrange Omeka content fields on the page. The Civil War Governors theme includes a header with rotating historical images from the Library of Congress, a navigation menu, and KHS branding. The footer has linked graphic logos of major funders of CWGK work. The custom homepage allows users to encounter the documents in a variety of ways—serving as an advertisement for the site's functionality. The search bar allows simple keyword searches and has a link to the advanced search page (custom built by Dehner to take advantage of Solr capabilities). Users can also explore documents by genre—correspondence, petitions, etc.—or navigate through featured collections and exhibits, which can be rotated by staff through the back-end interface.

Design elements from the homepage continue throughout the site, including subtle gray horizontal lines with stars and the outline of the state of Kentucky. About and Reference pages were created using the HTML editor in the Simple Pages plugin. These pages are mostly text and straightforward graphics and fit within the design vision of the site with little more than the simple horizontal lines and the constant header and footer.

Exhibits—custom collections of uploaded Omeka items arranged and displayed with interpretative text using an intuitive editor—are easy to create and manage. To date, CWGK has expanded its blog series, classroom exercise, and podcast, The Caroline Chronicles, into a new networked interface of the identifiable people, places, and organizations found in her story. Planned exhibits will illustrate other curriculum materials and illustrate how CWGK set and maintains its documentary editing processes and policies. The first of these is to be a collection of documents which fell outside of the CWGK date or authorship scope for interesting reasons, showing how careful editors must be to follow their own rules and know when to apply them.

The CWGK staff thought most carefully about the item viewer and search pages, drawing on their experience with their own collection and also as researchers and users of other sites to make these interfaces—the heart of the digital documentary edition—make sense to its audiences.  The item viewer lays out the document facsimile PDF and the transcription side by side along with a combination metadata/citation/download column below the transcription. The metadata fields pull from Omeka's Dublin Core fields, the Chicago style citation is automatically built from the same, and the PDF download uses a customized plugin and additional PHP libraries to create an image branded with the unique document URL, the download date, and a permissions statement. CWGK was insistent on this branded PDF system to protect the property rights of its partner institutions and prevent raw images from being downloaded en masse. There were trade-offs in this arrangement, however. CWGK had to disable the pop-out functionality of its PDF viewer on the item page because the user could save a copy of the PDF from that enlarged window. Until CWGK is able to disable that download option within the PDF Embed plugin, users are still able to enlarge the document within the viewer box and download the branded PDF for detailed viewing.

Startup editions or those growing from beginnings as a personal or class project will likely want to develop a custom theme for their site as funding (or staff skills) allow. A significant reason that CWGK intends to stay with this Omeka site long-term is because of its attractive design and the ability to create complex and specific functionality with the use of stylesheets and plugins rather than the modification of core files. The important part to remember about investing funds or staff time in creating a custom theme for the site is that the majority of those costs are up-front. After the basic structure of the site is laid out, staff without detailed coding knowledge can upload items, add metadata, and create pages and exhibits on their own. The site can continue to grow as the edition grows in size and complexity without continual design help. That being said, careful attention should be paid during the design phase to what plugins and third-party tools the project wants to use immediately and which ones may need to come into play later on. The designer will not only have to create visual space for the tools to work (adding fields or buttons) but will also have to take care that core Omeka functions or other plugins do not interfere with one another.

Software and Server

The self-hosted version of Omeka runs on a Linux server, a common platform for open-source software development. Most of the state servers in Kentucky, however, run Windows. CWGK was able to set up a Linux virtual machine with the state IT office, but doing so required finding and working with a Linux specialist within that department and billing some extra labor hours to set up the virtual machine and install the software. This was all dealt with through a few days of preparation work before active development began and may not be a concern at different institutions (say, a university) with a more active development community.

Another issue common to all software but still important to potential Omeka adopters is maintaining interoperability over time. CWGK pieced together assorted plugins and third-party tools to achieve the functionality the project desired, but every update to the individual plugins, the Omeka core software, and the host server brings the risk of throwing one or more pieces out of alignment and breaking parts of the site. A website needs to be continually maintained, and projects should budget not simply for the baseline hosting fees but also reserve some funding for troubleshooting, too.

With these maintenance issues in mind, CWGK intends to create a locally accessible duplicate development instance of its site on the state server. This allows the project staff and developers to test solutions to problems before applying changes to the live site. In the future, this will be a testing ground for new features and tools so that the project can continue to serve its existing content to the public while strengthening its analytical capabilities.

The duplicate development instance is also a critical backup for the pages and exhibits that make up the CWGK site beyond the documents. While the Simple Pages and Exhibit Builder plugins make the creation of pages easy through a simple HTML toolbar and text box, there is no versioning capability on these pages. Changes are permanent, which is never a problem until it is.
Collections Management and Searching

The site continues to function well with over 10,000 items uploaded, and the project is confident that it will continue to do so as more go in. The collection structure is intuitive and useful for large, diverse projects that deal with multiple repositories. It took time to develop a procedure for batch importing the project's TEI-XML documents and their corresponding PDFs—including Dehner developing the new DropboxTEI plugin. After the pathway was created, however, CWGK has had few problems uploading new documents, nesting them within the proper collections, and mapping their metadata to the correct Dublin Core fields.

Though the system of grouping items into collections works well in most cases, CWGK has discovered some flaws. In the Omeka interface at the item-level view, the user is presented with links to next and previous documents.  A user might intuitively believe these links will provide access to the next or previous document in a collection or set of search results. However, Omeka links to the next or previous item by the automatic item number that is assigned when the item is added to the system. If the collection is not uploaded in its entirety, its items may be displayed out of order if users browse using these buttons. This is an issue which has sparked much discussion on the Omeka forums, particularly as it pertains to retaining order when paging through search results. We believe this is contrary to what users will expect and have elected to remove these links until we or the Omeka community can come up with a solution.

The search results page takes advantage of the Solr search plugin's faceting capacity. Faceting is a way of narrowing search results by metadata categories in the way that many e-commerce sites do. In fact, Solr is a popular tool for major online retailers. In the CWGK context, a user who is browsing or who has initiated a keyword search can limit their results by facets such as repository, collection, document genre, and place and date of creation. For many web users today, starting from a broad keyword search and narrowing results with facets to the correct set of items is intuitive. However, the CWGK staff had learned to construct complex initial queries using an advanced search page as users of earlier digital history sites. An advanced search page works in precisely the opposite way as a faceted search. Instead of starting broad and narrowing in, it works to create a complex query that delivers narrow results from the outset. Omeka's out-of-the-box search engine has this advanced search capability but not the ability to refine search results that Solr provides. So, Dehner designed an advanced search page that adapted Solr's engine to deliver a small, specific set of results from one or more metadata categories. CWGK users are thus able to discover items in ways that fit their personal preference and research needs.

Though the Solr Search plugin is designed to allow highlighted search terms in returned results, the CWGK team has not yet been able to make this functionality work on its site. We believe the issue can be resolved with modifications to the CWGK theme files to recognize and display the highlighted terms.

The ultimate goal of Civil War Governors of Kentucky is to create a digital research environment within which a user can encounter the past multi-dimensionally through the documents and the powerful annotation network that links the documents together. The next step is to deliver approximately 1,500 documents, annotated and set within dense social and geographic networks. These documents, the first of a projected 40,000, will demonstrate how CWGK will shape the ways researchers, students, and teachers will explore the past in the future. To this end, CWGK was awarded $62,400 in the May 2016 cycle of National Historical Publications & Records Commission (NHPRC) funding. This grant will run from October 2016 to September 2017. It will support that next phase of work: publishing an annotation interface of 1,500 fully edited and linked documents.

Because the basic elements of Omeka as a document viewing, collections management, and curatorial apparatus are unquestionably solid, CWGK  intends to build its annotation interface around the existing Omeka site rather than develop an entirely new interface. On one level, this is practical. The Omeka site is visually attractive and functional; there is little incentive to rebuild from the ground up in another platform.

Developing further CWGK products in Omeka means a handful of changes for the existing site. For one, the Early Access branding must be changed. The project had always intended to keep the Omeka site live as a curatorial and teaching platform after the full edition was completed and had decided to establish the URL as discovery.civilwarvgovernors.org rather than earlyaccess.civilwargovernors.org. Thus, when the first batch of annotated documents goes live in 2017, the site will become, simply, Discovery.

The existing 10,000 documents will by no means go away as the first 1,500 annotated documents roll out. Rather, CWGK will mark each of the 1,500 which feature an authoritative transcription and are linked together by the annotation plugin. If a user wants to query the entire database of annotated and not-yet-annotated documents, they can, but they will also be free to use the Solr search fields to limit their searches to only annotated documents or only the annotations themselves. Using the faceting features already built into the site, the user can browse the annotated set of documents as a separate (and constantly growing) edition, or can broaden their search to see those documents in the context of the full database.

Using funds from the NHPRC, CWGK is working with the developers at Brumfield Labs to develop an annotation management tool, called MashBill (read more here). This web application will allow editors and remote Graduate Research Associates to work with the published documents on the web. Users will highlight the names of people, organizations, places, and geographical features in the Omeka item view page, and MashBill will allow them to link those names to existing entity records or create new ones. MashBill will export unique TEI-XML files for each named entity, which can be uploaded into Omeka as individual items. MashBill will also insert reference tags into the TEI-XML of each document, and CWGK will modify the Omeka style sheet to recognize those reference tags as links to the appropriate entity item. Rather than annotations that are appended to each document, the CWGK database of people, organizations, places, and geographical features will live as unique items linked to documents and one another within the Omeka interface.

CWGK has an existing procedure for importing large batches of TEI-XML documents, which can be modified to fit the upload needs for the entity records. The more significant modifications will come in modifying the item view interface to recognize hyperlinks and display the annotations and their accompanying metadata.

The CWGK staff and the Kentucky Historical Society believe that the framework of this Omeka site is sufficiently robust and flexible to handle a planned 40,000 primary documents and hundreds of thousands of identifiable entity records. To preview what a small batch of annotated documents and their associated entity records may look like, visit the “Networking Caroline” exhibit.