Presentation for Comics and the Multimodal World


I want to thank everyone for this wonderful, and overdue, conference on the connections between comics and media technology. I approach things specifically from a digital humanities perspective, and so I thought I’d take this conference as an opportunity to start thinking about how DH and comics will impact the future of humanities education. What, in other words, are the digital humanities doing to comics scholarship and, conversely, what can comics (as a medium of communication) offer the digital humanities? I want to start briefly with this image by Nick Sousanis. Lots of people associate technology, and sometimes the digital humanities, with industrial modes of standardized production. That’s wrong on a number of levels, and I’m here to show you that it creates unnecessary anxiety and conflict.


First of all there is more than a little anxiety over this term “digital humanities.” I’ve had several of my colleagues start digital humanities centers and ask students and professors on campus what kinds of workshops they’d like to have presented there. Almost without fail, the most popularly requested workshop is “Introduction to the Digital Humanities.” People assume that DH is the “next big thing,” like queer theory or deconstruction, and want to know how to conceptualize it. But DH isn’t really a discipline or a field. I like what Ted Underwood says about DH, that it is a loose constellation of digital practices that really aren’t organized in any particular way — and there is no reason to think that it will be a field in 10 years. Similarly, Bill Pannapacker’s discussion of “big tent DH” is useful here: i.e. I consider DH to be anything that combines humanities study with digital methodologies, pedagogies, or subjects. Consider also the website, which loads a different definition of the digital humanities each time you refresh the page. Still there have been, and continue to be, some important experiments that work between comics and digital technologies. Here are a few of them.


Probably the most well-known Comics/DH project is Lev Manovich and Jeremy Douglass’s cultural analytics of 1 million manga pages. In the article, Manovich produces a visual reading practice similar to that of Franco Moretti’s distant reading, which in his words “would allow us to understand which manga series are most typical and which are most unique stylistically; to find all series where graphical language significantly changes over time […]; to investigate if shorter series and longer series have different patterns; to seperate the artists who significantly vary their graphical languages from series to series from the artists who do not; etc” (4). As the eye glances across the page, it picks out many different kinds of visual similarities – from shading to color to line patterns. Manovich, thus takes from distant reading and topic modeling practices a visual form of stylistic analysis that is possible only with a large quantity of visual references graphed in ways like the one pictured here.


A second important form of analysis that has been linked to comics from the digital humanities field is the implementation and discussion of TEI markup language. TEI is a set of XML standards that represents texts in digital form, and is commonly used in digital archives when scholars want to repurpose a text. For example,  Rachel Lee and J. Alexandra McGee explain that TEI allows almost limitless possibilities for tagging a text. “There are tags to describe the physical object, such as its current location, material, dimensions, watermarks, ink color, handwriting, and damage. XML elements can map the text’s content through rhyme schemes, grammatical structure, technical language, or foreign expressions. Projects concerned with linguistics can encode parts of speech, while others may choose to include the GPS locations of place names mentioned in the text or link to other relevant material available on the web. Transcribers can include alternate readings for an unclear word, and translators can use linking structures to provide multiple translations for a single line, stanza, or entire poem.” Tags make the text more searchable, as well as visualizable, in ways that are limited only by the descriptions in the tags. Given all of the semantic possibilities of linguistic text, what could we do with a series of panels like this one? (What’s the TEI description of a Kirby Dot?)


John Walsh’s groundbreaking essay on “Comic Book Markup Language: An Introduction and Rationale” articulates some of these complications to traditional forms of TEI markup when faced with sequential art. For example, he cites these two panels from The Incredible Hulk #114. “While many design features common to textual documents, such as text size and font characteristics, may be reasonably and useful described using common TEI techniques,” Walsh argues, “it would be futile and impractical to attempt to describe every detail of every picture in a comic book document. The encoded document, with markup containing and describing metadata, structure, transcription, and analysis, should co-exist with and be linked to digital facsimile page images of the comic book.” Describing what happens in a comic to ourselves is difficult, describing what happens in a comic to a machine, even more so.


Things get even more complicated (and interesting) when digital technology allows scholars to begin to experiment with the comic form in their scholarship. This is one of the most memorable submissions to The Comics Grid that I was allowed to peer-review in the last year. It is Nichlas Labarre’s modular and visual analysis of Ronald Reagan’s wink in The Dark Knight Returns. Labarre decided to present his analysis in a way that encouraged digressions, while also allowing the essay to show the reader a visual comparison of Reagan’s wink (in the comic as well as in political cartoons from the 80s) and Superman’s famous wink (in the comic as well as Silver Age Superman stories) as a way of underscoring the satire of DKR. The modular nature of Labarre’s presentation would be even more complicated for to translate into a TEI structure for encoding — yet it is also the ability of such a text to be digitally produced and presented that allows it to exist in the first place.


As we can see, comic books provide some tremendous challenges to the textual and computational modalities of coding and programming by simply combining the visual and the verbal. Comics also force us to ask some questions of machine-reading. What does it mean, for example, for a machine to view or interpret a set of sequential images? This page by Nick Sousanis, who is currently completing the first dissertation entirely written as a comic book,  shows other complications. Visual elements can semiotically call forth different cultural styles (manga, realism, french science fiction), they can present temporally sequential panels that are also meant to be spatially read as simultaneous, they can construct meaning that’s meant to be read between the images and the words on the page. Now all of this is obvious to anyone who’s read Scott McCloud or Therry Groenstein, but they have huge implications to programming languages and data-mining experiments which are used to parsing text and numbers.



The problem is, of course, the relationship between images, text, and code. If you look at the idea of a pixel (coined probably earlier than 1932 as a portmanteau between picture (pix) and element (el)), it is conceptualized as the smallest visual unit that is used to create digital pictures. Any digital image is constructed out of a number of pixels. Pixels are, themselves made up by computer bytes. And the number of bytes per pixel give pixels different color possibilities. For 1 bpp, for example, allows a byte to be on or off (binary) and the resulting image is monochromatic. So-called “True-color,” pixelated images that give the closest digitally-represented images to real life, have 24 bites per pixel or 16.8 million possible colors – and you can imagine the number of on/off combinations that create specific, discrete colors. The growing disparity between numbers of bytes and pixels means that pixels have largely come to “represent” bytes of memory rather than — as George Dyson has argued about the early days of computing — directly be them. This means that the .jpg image I’m showing here (which is made up of 4272 x 2848 pixels) represents about 12.1 million bytes or 292 million possible colors derived from the on/off modulation of those bytes. So, imagine for a moment, what a machine reads when it translates and then displays this picture: a string of binary code that represents specific instructions for displaying particular combinations of pixels. This is the complexity found simply in a computational display of a specific image, let alone anything like comic book analogues to distant reading practices such as Topic Modeling and Natural Language Processing – fields that are only now being explored by figures like Stephen Ramsay, Matthew Jockers, and Ted Underwood. Right now, while it is certainly possible to create the kinds of collages imagined by Manovich and Douglass and allow human eyes to derive patterns, imagine what more sophisticated methods of distant reading would mean for the verbal/visual combinations found in comics.


This is all the more reason for encouraging scholarly experimentation in the comics form. Anastasia Salter and I are editing a special issue of Digital Humanities Quarterly on “Comics as Scholarship.” Our purpose is to encourage scholars to think about the modal affordances of the comic book for research. So far, apart from submissions by Nick Sousanis and Ulysees Seen! creator Robert Berry and interviewer Gene Kannenberg, Jr., we’ve also had some interesting experiments by my co-panelist Aaron Kashtan, Franny Howes, Liz Losh and John Alexander, BJ Parker, Jason Helms, Robert Watkins and Tom Lindsley, and Aaron Humphrey. In a larger sense, the digital possibilities inherent with increased computation and visual display allow both scholars and students to experiment with scholarly inquiry in a way that was simply not possible even five years ago. I also feel that — considering the narrative capabilities of different kinds of panel layouts — taking the visual and verbal language of comics seriously will only increase the potential for new realms of analysis.


Adeline Koh and Roopsi Risam use bitstrips to create brief on race, gender, and class issues in the digital humanities community on their website Postcolonial Digital Humanities. On one level, this strip talks about the new forms of inequity emerging around technology. Take, for instance, the revelations about the FoxConn Plant last year in Shenzen, China. On another level, control is also situated around who has the ability to manipulate the graphical user interfaces (GUIs) that help non-coders interface with technology. Is it important — as I would contend — for humanities instructors to become more acquainted with the “guts” of the computer, program microprocessors like Arduinos, and give students a hands-on introduction to manipulating technology? Is it important to champion open-source software that gives everyone the chance to have control over the programs that they use? In the context of comics, would a better understanding of comic design and sequential language give us another tool to use for communication, visualization, and analysis? These are all open questions, but I’d argue, they are questions that are emerging as we start to take seriously the connection between comics and the digital humanities.



Leave a Reply