University of Michigan School of Information faculty and PhD students are creating and sharing knowledge that helps build a better world. Here are some of their recent publications. 

Publications 

Constructing risk in trustworthy digital repositories

Journal of Documentation, August 2024 

Rebecca D. Frank

Purpose: This article investigates the construction of risk within trustworthy digital repository audits. It contends that risk is a social construct, and social factors influence how stakeholders in digital preservation processes comprehend and react to risk. 

Design/methodology/approach: This research employs a qualitative research design involving in-depth semi-structured interviews with stakeholders in the Trustworthy Digital Repository Audit and Certification (TRAC) process, and document analysis of the TRAC checklist and audit reports. I apply an analytic framework based on the Model for the Social Construction of Risk in Digital Preservation to this data. 

Findings: The findings validate the argument that risk in digital preservation is indeed socially constructed and demonstrate that the eight factors in the Model for the Social Construction of Risk in Digital Preservation do indeed influence how stakeholders constructed their understanding of risk. Of the eight factors in the model, communication, expertise, uncertainty and vulnerability were found to be the most influential in the construction of risk during the TRAC audit process. The influence of complexity, organizations political culture, were more limited. 

Originality/value: This article brings new insights to digital preservation by demonstrating the importance of understanding risk as a social construct. I argue that risk identification and/or assessment is only the first step in the long-term preservation of digital information and show that perceptions of risk in digital preservation are shaped by social factors by applying theories of social construction and risk perception to an analysis of the TRAC process.

An empirical examination of data reuser trust in a digital repository

Journal of the Association for Information Science and Technology, August 2024

Elizabeth Yakel, Ixchel M. Faniel, Lionel P. Robert Jr

Most studies of trusted digital repositories have focused on the internal factors delineated in the Open Archival Information System (OAIS) Reference Model—organizational structure, technical infrastructure, and policies, procedures, and processes. Typically, these factors are used during an audit and certification process to demonstrate a repository can be trusted. The factors influencing a repository's designated community of users to trust it remains largely unexplored. This article proposes and tests a model of trust in a data repository and the influence trust has on users' intention to continue using it. Based on analysis of 245 surveys from quantitative social scientists who published research based on the holdings of one data repository, findings show three factors are positively related to data reuser trust—integrity, identification, and structural assurance. In turn, trust and performance expectancy are positively related to data reusers' intentions to return to the repository for more data. As one of the first studies of its kind, it shows the conceptualization of trusted digital repositories needs to go beyond high-level definitions and simple application of the OAIS standard. Trust needs to encompass the complex trust relationship between designated communities of users that the repositories are being built to serve.

A mobile health intervention for emerging adults with regular cannabis use: A micro-randomized pilot trial design protocol

Contemporary Clinical Trials, August 2024 

Lara N. Coughlin, Maya Campbell, Tiffany Wheeler, Chavez Rodriguez, Autumn Rae Florimbio, Susobhan Ghosh, Yongyi Guo, Pei-Yao Hung, Mark W. Newman, Huijie Pan, Kelly W. Zhang, Lauren Zimmermann, Erin E. Bonar, Maureen Walton, Susan Murphy, Inbal Nahum-Shani

Background: Emerging adult (EA) cannabis use is associated with increased risk for health consequences. Just-in-time adaptive interventions (JITAIs) provide potential for preventing the escalation and consequences of cannabis use. Powered by mobile devices, JITAIs use decision rules that take the person's state and context as input, and output a recommended intervention (e.g., alternative activities, coping strategies). The mHealth literature on JITAIs is nascent, with additional research needed to identify what intervention content to deliver when and to whom.

Methods: Herein we describe the protocol for a pilot study testing the feasibility and acceptability of a micro-randomized trial for optimizing MiWaves mobile intervention app for EAs (ages 18–25; target N = 120) with regular cannabis use (≥3 times per week). Micro-randomizations will be determined by a reinforcement learning algorithm that continually learns and improves the decision rules as participants experience the intervention. MiWaves will prompt participants to complete an in-app twice-daily survey over 30 days and participants will be micro-randomized twice daily to either: no message or a message [1 of 6 types varying in length (short, long) and interaction type (rate message, rate message + click additional resources, rate message + fill in the blank/select an option)]. Participants recruited via social media will download the MiWaves app, and complete screening, baseline, weekly, post-intervention, and 2-month follow-up assessments. Primary outcomes include feasibility and acceptability, with additional exploratory behavioral outcomes.

Conclusion: This study represents a critical first step in developing an effective mHealth intervention for reducing cannabis use and associated harms in EAs.

Teachable moments: TikTok social drama as a site of Blackfeminist intellectual production

Information, Communication, and Society, July 2024 

Chelsea Peterson-Salahuddin

This paper mobilizes Turner’s concept of ‘social drama’ – dramatic social events marked by conflict, antagonism, and competition through which communities articulate social norms – to examine how users engage in drama to negotiate the intersecting race,gender, and sexual power dynamics on TikTok. Through an analysis of two TikTok social dramas, I demonstrate how the technological affordances of the platform shape how users respond to drama and argue TikTok’s predominantly younger use-base utilize the platform to discuss, negotiate, and put in place socially progressive norms around racism, sexism, and homophobia. Further, I show how these dramas create opportunities for Black and of color women, femme, and queerTikTok users to engage in digital Black feminism by drawing on their lived experiences and using platform affordances to comment on the situation and educate other users. The paper concludes by considering the role of platform social dramas in bringing both opportunity and harm to marginalized users.

A Danish Blend: The Copenhagen Walkshop

DIS ‘24 Companion: Companion Publication of the 2024 ACM Designing Interactive Systems Conference, July 2024 

Andrea Resmini, Dan Klyn, Bertil Lindenfalk, Miriam Tedeschi, Daniel Szuc, Josephine Wong

The walkshop introduces participants to an embodied, structural approach to understand, conceptualize, and design experiences that blend physical and digital space to create a novel space of action, with its own sense of presence, its own afordances, and its very special challenges. It consists of an outdoors morning walking and exploratory session, and of an afternoon mapping and refective session at the conference venue. During the morning session, participants directly experience the urban fabric of Copenhagen and engage in activities meant to explore and expose the way digital and physical space commingle and become a layered blended space. During the afternoon session, the participants turn notes and observations into maps with the help of methods and tools provided by the facilitators. Attention is paid to identifying friction between pace layers and to the structure, participating elements, and relationships that support the experience in either digital, physical, or blended space, and to refect on how the structures of embodiment and spatiality shape experience and act as important, non-interface level grounding elements in the design of human activity in all types of space. The walkshop concludes with a plenary discussion of the deliverables created by participants, what insights were gained in the process, and possible developments to follow.

What to Expect When You’re Accessing: An Exploration of User Privacy Rights in People Search Websites

Proceedings on Privacy Enhancing Technologies Symposium, July 2024

Kejsi Take, Jordyn Young, Rasika Bhalerao, Kevin Gallagher, Andrea Forte, Damon McCoy, Rachel Greenstadt

People Search Websites, a category of data brokers, collect, catalog, monetize and often publicly display individuals’ personally identifiable information (PII). We present a study of user privacy rights in 20 such websites assessing the usability of data access and data removal mechanisms. We combine insights from these two processes to determine connections between sites, such as shared access mechanisms or removal effects. We find that data access requests are mostly unsuccessful. Instead, sites cite a variety of legal exceptions or misinterpret the nature of the requests. By purchasing reports, we find that only one set of connected sites provided access to the same report they sell to customers. We leverage a multiple step removal process to investigate removal effects between suspected connected sites. In general, data removal is more streamlined than data access, but not very transparent; questions about the scope of removal and reappearance of information remain. Confirming and expanding the connections observed in prior phases, we find that four main groups are behind 14 of the sites studied, indicating the need to further catalog these connections to simplify removal

Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course

ICER ‘24: Proceedings of the 2024 ACM Conference on International Computing Education Research, July 2024 

Aadarsh Padiyath, Xinying Hou, Amy Pang, Diego Viramontes Vargas, Xingjian Gu, Tamara Nelson-Fromm, Zihan Wu, Mark Guzdial, Barbara Ericson 

The capability of large language models (LLMs) to generate, debug, and explain code has sparked the interest of researchers and educators in undergraduate programming, with many anticipating their transformative potential in programming education. However, decisions about why and how to use LLMs in programming education may involve more than just the assessment of an LLM’s technical capabilities. Using the social shaping of technology theory as a guiding framework, our study explores how students’ social perceptions influence their own LLM usage. We then examine the correlation of self-reported LLM usage with students’ self-efficacy and midterm performances in an undergraduate programming course. Triangulating data from an anonymous end-of-course student survey (n = 158), a mid-course self-efficacy survey (n=158), student interviews (n = 10), self-reported LLM usage on homework, and midterm performances, we discovered that students’ use of LLMs was associated with their expectations for their future careers and their perceptions of peer usage. Additionally, early self-reported LLM usage in our context correlated with lower self-efficacy and lower midterm scores, while students’ perceived over-reliance on LLMs, rather than their usage itself, correlated with decreased self-efficacy later in the course

712forTask7 at #SMM4H 2024 Task 7: Classifying Spanish Tweets Annotated by Humans versus Machines with BETO Models

Proceedings of The 9th Social Media Mining for Health Research and Applications (SMM4H 2024) Workshop and Shared Tasks, August 2024 

Hafizh R. Yusuf, David Belmonte, Dalton Simancek, V.G.Vinod Vydiswaran

The goal of Social Media Mining for Health (#SMM4H) 2024 Task 7 was to train a machine learning model that is able to distinguish between annotations made by humans and those made by a Large Language Model (LLM). The dataset consisted of tweets originating from #SMM4H 2023 Task 3, wherein the objective was to extract COVID-19 symptoms in LatinAmerican Spanish tweets. Due to the lack of additional annotated tweets for classification, we reframed the task using the available tweets and their corresponding human or machine annotator labels to explore differences between the two subsets of tweets. We conducted an exploratory data analysis and trained a BERT-based classifier to identify sampling biases between the two subsets. The exploratory data analysis found no significant differences between the samples and our best classifier achieved a precision of 0.52 and a recall of 0.51, indicating near-random performance. This confirms the lack of sampling biases between the two sets of tweets and is thus a valid dataset for a task designed to assess the authorship of annotations by humans versus machines.

LHS712_ADENotGood at #SMM4H 2024 Task 1: Deep-LLMADEminer: A deep learning and LLM pharmacovigilance pipeline for extraction and normalization of adverse drug event mentions on Twitter

Proceedings of The 9th Social Media Mining for Health Research and Applications (SMM4H 2024) Workshop and Shared Tasks, August 2024 

Yifan Zheng, Jun Gong, Shushun Ren, Dalton Simancek, V.G.Vinod Vydiswaran

Adverse drug events (ADEs) pose major public health risks, with traditional reporting systems often failing to capture them. Our proposed pipeline, called Deep-LLMADEminer, used natural language processing approaches to tackle this issue for #SMM4H 2024 shared task 1. Using annotated tweets, we built a three part pipeline: RoBERTa for classification, GPT4-turbo for span extraction, and BioBERT for normalization. Our models achieved F1-scores of 0.838, 0.306, and 0.354, respectively, offering a novel system for Task 1 and similar pharmacovigilance tasks

LHS712NV at #SMM4H 2024 Task 4: Using BERT to classify Reddit posts on non-medical substance use

Proceedings of The 9th Social Media Mining for Health Research and Applications (SMM4H 2024) Workshop and Shared Tasks, August 2024

Valeria Fraga, Neha Nair, Dalton Simancek. V.G.Vinod Vydiswaran

This paper summarizes our participation in the Shared Task 4 of #SMM4H 2024. Task 4 was a named entity recognition (NER) task identifying clinical and social impacts of non-medical substance use in English Reddit posts. We employed the Bidirectional Encoder Representations from Transformers (BERT) model to complete this task. Our team achieved an F1-score of 0.892 on a validation set and a relaxed F1- score of 0.191 on the test set.

Enhancing healthcare accessibility through telehealth for justice impacted individuals

Frontiers in Public Health, August 2024

Karmen S. Williams, Marianna J. Singh, Johanna E. Elumn, Megan Threats, Yongjie Sha, Terika McCall, Karen Wang, Bria Massey, Mary L. Peng, Kevin Wiley

Telehealth is a great tool that makes accessing healthcare easier for those incarcerated and can help with reentry into the the community. Justice impacted individuals face many hardships including adverse health outcomes which can be mitigated through access to telehealth services and providers. During the federally recognized COVID-19 pandemic the need for accessible healthcare was exacerbated and telehealth use surged. While access to telehealth should be considered a necessity, there are many challenges and barriers for justice impacted individuals to be able to utilize this service. This perspective examines aspects of accessibility, pandemic, policy, digital tools, and ethical and social considerations of telehealth in correctional facilities. Carceral facilities should continue to innovate and invest in telehealth to revolutionize healthcare delivery, and improve health outcomes for justice impacted individuals.

What is research data “misuse”? And how can it be prevented or mitigated?

Journal of the Association for Information Science and Technology, July 2024

Irene V. Pasquetto, Zoë Cullen, Andrea Thomer, Morgan Wofford

Despite increasing expectations that researchers and funding agencies release their data for reuse, concerns about data misuse hinder the open sharing of data. The COVID-19 crisis brought urgency to these concerns, yet we are currently missing a theoretical framework to understand, prevent, and respond to research data misuse. In the article, we emphasize the challenge of defining misuse broadly and identify various forms that misuse can take, including methodological mistakes, unauthorized reuse, and intentional misrepresentation. We pay particular attention to underscoring the complexity of defining misuse, considering different epistemological perspectives and the evolving nature of scientific methodologies. We propose a theoretical framework grounded in the critical analysis of interdisciplinary literature on the topic of misusing research data, identifying similarities and differences in how data misuse is defined across a variety of fields, and propose a working definition of what it means “to misuse” research data. Finally, we speculate about possible curatorial interventions that data intermediaries can adopt to prevent or respond to instances of misuse

Investigating Metacognitive Behaviors with Online Learning Support Tools

IEEE ICALT, July 2024

Masanori Yamada, Xuewang Geng, Yoshiko Goda, Stephanie Teasley 

As information technology advanced, accuracy of technology-driven assessment is being improved. In order to assess learning performance and awareness, assessment of metacognition level with technology can be useful to understand learner's learning comprehension and awareness. Metacognition is one of the most important elements for successful learning. However, current way to evaluate metacognition level focuses on psychological method such as questionnaire and interview. The recent growth of learning analytics research has demonstrated the relationships between metacognition, learning awareness, and learning behaviors. This study aims to investigate metacognitive learning behaviors using small grain data on eBook and learning analytics dashboard (LAD) over eight weeks in a university course. To do so, we determined high and low metacognitive learner groups using the Metacognitive Awareness Inventory and investigated the differences between the two groups in eBook and LAD. The findings suggest that four learning behaviors eBook and LAD were detected as metacognitive learning behaviors, and contribute to the improvement of technology-driven assessment.

AI governance through fractal scaling: integrating universal human rights with emergent self-governance for democratized technosocial systems

AI & Society, July 2024

Ron. Eglash, Micheal Nayebare, Kwame. Robinson, Lionel Robert, A. Bennett, U. Kimanuka, C. Maina 

One of the challenges facing AI governance is the need for multiple scales. Universal human rights require a global scale. If someone asks AI if education is harmful to women, the answer should be “no” regardless of their location. But economic democratization requires local control: if AI’s power over an economy is dictated by corporate giants or authoritarian states, it may degrade democracy’s social and environmental foundations. AI democratization, in other words, needs to operate across multiple scales. Nature allows the multiscale flourishing of biological systems through fractal distributions. In this paper, we show that key elements of the fractal scaling found in nature can be applied to the AI democratization process. We begin by looking at fractal trees in nature and applying similar analytics to tree representations of online conversations. We first examine this application in the context of OpenAI’s “Democratic Inputs” projects for globally acceptable policies. We then look at the advantages of independent AI ownership at local micro-levels, reporting on initial outcomes for experiments with AI and related technologies in community-based systems. Finally, we offer a synthesis of the two, micro and macro, in a multifractal model. Just as nature allows multifractal systems to maximize biodiverse flourishing, we propose a combination of community-owned AI at the micro-level, and globally democratized AI policies at the macro-level, for a more egalitarian and sustainable future.

Causal Pattern Diagrams in Science Texts Support Explanation

International Society of the Learning Sciences, July 2024 

Hari Subramonyam, Eytan Adar, Priti Shah, Colleen M. Seifert

Understanding causal models in science texts requires learners to construct a mental model of complex causal relationships despite unfamiliar vocabulary and limited knowledge. Prior studies show presenting diagrams helps learners improve, but little is known about how learning improves with causal diagrams. In an online study, we tested adults’ explanations of five science phenomena after reading text with or without a schematic causal diagram. The quantitative and qualitative results show causal pattern diagrams promote holistic quality in learners' explanations. Providing visualizations of complex causal relationships within science texts can scaffold learning by highlighting the need to understand causal patterns in science. 

Augmenting clinicians’ analytical workflow through task-based integration of data visualizations and algorithmic insights: a user-centered design study 

Journal of the American Medical Informatics Association, July 2024

Till Scholich,  Shriti Raj, Joyce Lee, Mark W Newman

Objectives: To understand healthcare providers’ experiences of using GlucoGuide, a mockup tool that integrates visual data analysis with algorithmic insights to support clinicians’ use of patientgenerated data from Type 1 diabetes devices.

Materials and Methods: This qualitative study was conducted in three phases. In Phase 1, 11 clinicians reviewed data using commercial diabetes platforms in a think-aloud data walkthrough activity followed by semistructured interviews. In Phase 2, GlucoGuide was developed. In Phase 3, the same clinicians reviewed data using GlucoGuide in a think-aloud activity followed by semistructured interviews. Inductive thematic analysis was used to analyze transcripts of Phase 1 and Phase 3 think-aloud activity and interview.

Results: 3 high level tasks, 8 sub-tasks, and 4 challenges were identified in Phase 1. In Phase 2, 3 requirements for GlucoGuide were identified. Phase 3 results suggested that clinicians found GlucoGuide easier to use and experienced a lower cognitive burden as compared to the commercial diabetes data reports that were used in Phase 1. Additionally, GlucoGuide addressed the challenges experienced in Phase 1.

Discussion: The study suggests that the knowledge of analytical tasks and task-specific visualization strategies in implementing features of data interfaces can result in tools that lower the perceived burden of engaging with data. Additionally, supporting clinicians in contextualizing algorithmic insights by visual analysis of relevant data can positively influence clinicians’ willingness to leverage algorithmic support.

Conclusion: Task-aligned tools that combine multiple data-driven approaches, such as visualization strategies and algorithmic insights, can improve clinicians’ experience in reviewing device data.

The life cycle of large language models in education: A framework for understanding sources of bias

British Journal of Educational Technology, July 2024

Jinsook Lee, Yann Hicke, Renzhe Yu, Christopher Brooks, Rene F. Kizilcec

Large language models (LLMs) are increasingly adopted in educational contexts to provide personalized support to students and teachers. The unprecedented capacity of LLM-based applications to understand and generate natural language can potentially improve instructional effectiveness and learning outcomes, but the integration of LLMs in education technology has renewed concerns over algorithmic bias, which may exacerbate educational inequalities. Building on prior work that mapped the traditional machine learning life cycle, we provide a framework of the LLM life cycle from the initial development of LLMs to customizing pre-trained models for various applications in educational settings. We explain each step in the LLM life cycle and identify potential sources of bias that may arise in the context of education. We discuss why current measures of bias from traditional machine learning fail to transfer to LLM-generated text (eg, tutoring conversations) because text encodings are high-dimensional, there can be multiple correct responses, and tailoring responses may be pedagogically desirable rather than unfair. The proposed framework clarifies the complex nature of bias in LLM applications and provides practical guidance for their evaluation to promote educational equity.

Profile update: the effects of identity disclosure on network connections and language

EPJ Data Science, June 2024 

Minje Choi,  Daniel M. Romero, David Jurgens

Our social identities determine how we interact and engage with the world surrounding us. In online settings, individuals can make these identities explicit by including them in their public biography, possibly signaling a change in what is important to them and how they should be viewed. While there is evidence suggesting the impact of intentional identity disclosure in online social platforms, its actual effect on engagement activities at the user level has yet to be explored. Here, we perform the first large-scale study on Twitter that examines behavioral changes following identity disclosure on Twitter profiles. Combining social networks with methods from natural language processing and quasi-experimental analyses, we discover that after disclosing an identity on their profiles, users (1) tweet and retweet more in a way that aligns with their respective identities, and (2) connect more with users that disclose similar identities. We also examine whether disclosing the identity increases the chance of being targeted for offensive comments and find that in fact (3) the combined effect of disclosing identity via both tweets and profiles is associated with a reduced number of offensive replies from others. Our findings highlight that the decision to disclose one’s identity in online spaces can lead to substantial changes in how they express themselves or forge connections, with a lesser degree of negative consequences than anticipated.

Proposing a Context-informed Layer-based Framework: Incorporating Context into Designing mHealth Technology for Fatigue Management

DIS ‘24: Proceedings of the 2024 ACM Designing Interactive Systems Conference, July 2024 

Xinghui (Erica) Yan, Loubna Baroudi, Rongqi Bei, Leila Boudalia, Stephen M Cain, Kira Barton, Mark W. Newman

Owing to the multifactorial nature of fatigue, leveraging context to effectively monitor and intervene with fatigue symptoms presents a significant challenge. This paper aimed to understand how to incorporate context into designing mHealth systems for fatigue management. We conducted a two-week field study with 20 fatigue vulnerable individuals using an activity-tracking sensor and self reporting. We conducted data-prompted interviews to explore phenomena about participants’ fatigue experiences. Findings show a heterogeneous relationship between context and fatigue, which can be attributed to the phenomena that: (1) participants were influenced by multiple fatigue-inducing factors for different durations; (2) broad contexts moderated participants’ perceptions and coping strategies in response to local contexts; (3) the predictability and repetition of activities influenced participants’ fatigue perception and coping strategies. We propose a context-informed layer-based framework integrating these phenomena and discuss implications for designing fatigue management tools informed by our framework.

Making Trouble: Techniques for Queering Data and AI Systems

DIS ‘24 Companion: Companion Publication of the 2024 ACM Designing Interactive Systems Conference, July 2024 

Anh-Ton Tran, Annabel Rothschild, Kay Kender, Ekat Osipova, Brian Kinnee, Jordan Taylor, Louie Søs Meyer, Oliver L. Haimson, Ann Light, Carl DiSalvo

This one day workshop will explore queering as a design technique for troubling data and AI systems, ranging from quotidian personal data to recent Generative AI tools. By surfacing numerous instances of queering data or AI, we will come together to develop an archive of techniques for queering or artful subversion. From this archive, participants will select a technique and develop a speculative prototype or artifact via critical making. In doing so, we resist techno-determinism and conventional narratives of AI harms and benefits by tracing queer possibilities outside these categories

Can LLMs Effectively Leverage Graph Structural Information through Prompts in Text-Attributed Graphs, and Why?

Transactions on Machine Learning Research, June 2024 

Jin Huang, Xingjian Zhang, Qiaozhu Mei, Jiaqi Ma

Large language models (LLMs) are gaining increasing attention for their capability to process graphs with rich text attributes, especially in a zero-shot fashion. Recent studies demonstrate that LLMs obtain decent text classification performance on common text-rich graph benchmarks, and the performance can be improved by appending encoded structural information as natural languages into prompts. We aim to understand why the incorporation of structural information inherent in graph data can improve the prediction performance of LLMs. First, we rule out the concern of data leakage by curating a novel leakage-free dataset and conducting a comparative analysis alongside a previously widely-used dataset. Second, as past work usually encodes the ego-graph by describing the graph structure in natural language, we ask the question: do LLMs understand the prompts in graph structures? Third, we investigate why LLMs can improve their performance after incorporating structural information. Our exploration of these questions reveals that (i) there is no substantial evidence that the performance of LLMs is significantly attributed to data leakage; (ii) instead of understanding prompts as graph structures, LLMs tend to process prompts more as contextual paragraphs and (iii) the most efficient elements of the local neighborhood included in the prompt are phrases that are pertinent to the node label, rather than the graph structure. 

Pre-prints, Working Papers, Articles, Reports, Workshops and Talks

The Role of Network and Identity in the Diffusion of Hashtags 

arXiv, July 2024 

Aparna Ananthasubramaniam, Yufei “Louise” Zhu, David Jurgens, Daniel M. Romero

Although the spread of behaviors is influenced by many social factors, existing 8 literature tends to study the effects of single factors—most often, properties of the 9 social network—on the final cascade. In order to move towards a more integrated 10 view of cascades, this paper offers the first comprehensive investigation into the 11 role of two social factors in the diffusion of 1,337 popular hashtags representing the 12 production of novel culture on Twitter: 1) the topology of the Twitter social network 13 and 2) performance of each user’s probable demographic identity. Here, we show 14 that cascades are best modeled using a combination of network and identity, rather 15 than either factor alone. This combined model best reproduces a composite index 16 of ten cascade properties across all 1,337 hashtags. However, there is important 17 heterogeneity in what social factors are required to reproduce different properties 18 of hashtag cascades. For instance, while a combined network+identity model best 19 predicts the popularity of cascades, a network-only model has better performance 20 in predicting cascade growth and an identity-only model in adopter composition. 21 We are able to predict what type of hashtag is best modeled by each combination 22 of features and use this to further improve performance. Additionally, consistent 23 with prior literature on the combined network+identity model most outperforms the 24 single-factor counterfactuals among hashtags used for expressing racial or regional 25 identity, stance-taking, talking about sports, or variants of existing cultural trends 26 with very slow- or fast-growing communicative need. In sum, our results imply the 27 utility of multi-factor models in predicting cascades, in order to account for the 28 varied ways in which network, identity, and other social factors play a role in the 29 diffusion of hashtags on Twitter

VALUESCOPE: Unveiling Implicit Norms and Values via Return Potential Model of Social Interactions

arXiv, July 2024

Chan Young Park, Shuyue Stella Li, Hayoung Jung, Svitlana Volkova, Tanushree Mitra, David Jurgens, Yulia Tsvetkov

This study introduces VALUESCOPE, a framework leveraging language models to quantify social norms and values within online communities, grounded in social science perspectives on normative structures. We employ VALUESCOPE to dissect and analyze linguistic and stylistic expressions across 13 Reddit communities categorized under gender, politics, science, and finance. Our analysis provides a quantitative foundation showing that even closely related communities exhibit remarkably diverse norms. This diversity supports existing theories and adds a new dimension—community preference—to understanding community interactions. VALUESCOPE not only delineates differing social norms among communities but also effectively traces their evolution and the influence of significant external events like the U.S. presidential elections and the emergence of new sub-communities. The framework thus highlights the pivotal role of social norms in shaping online interactions, presenting a substantial advance in both the theory and application of social norm studies in digital spaces.

Seeing Like an AI: How LLMs Apply (and Misapply) Wikipedia Neutrality Norms

arXiv, July 2024

Joshua Ashkinaze, Ruijia Guan, Laura Kurek, Eytan Adar, Ceren Budak, Eric Gilbert

Large language models (LLMs) are trained on broad corpora and then used in communities with specialized norms. Is providing LLMs with community rules enough for models to follow these norms? We evaluate LLMs’ capacity to detect (Task 1) and correct (Task 2) biased Wikipedia edits according to Wikipedia’s Neutral Point of View (NPOV) policy. LLMs struggled with bias detection, achieving only 64% accuracy on a balanced dataset. Models exhibited contrasting biases (some under- and others over-predicted bias), suggesting distinct priors about neutrality. LLMs performed better at generation, removing 79% of words removed by Wikipedia editors. However, LLMs made additional changes beyond Wikipedia editors’ simpler neutralizations, resulting in high-recall but low-precision editing. Interestingly, crowdworkers rated AI rewrites as more neutral (70%) and fluent (61%) than Wikipedia-editor rewrites. Qualitative analysis found LLMs sometimes applied NPOV more comprehensively than Wikipedia editors but often made extraneous non-NPOV-related changes (such as grammar). LLMs may apply rules in ways that resonate with the public but diverge from community experts. While potentially effective for generation, LLMs may reduce editor agency and increase moderation workload (e.g., verifying additions). Even when rules are easy to articulate, having LLMs apply them like community members may still be difficult.

Crowdsourced reviews reveal substantial disparities in public perceptions of parking

arXiv, July 2024

Lingyao Li, Songhua Hu, Ly Dinh, Libby Hemphill

Due to increased reliance on private vehicles and growing travel demand, parking remains a longstanding urban challenge globally. Quantifying parking perceptions is paramount as it enables decision-makers to identify problematic areas and make informed decisions on parking management. This study introduces a cost-effective and widely accessible data source, crowdsourced online reviews, to investigate public perceptions of parking across the U.S. Specifically, we examine 4,987,483 parking-related reviews for 1,129,460 points of interest (POIs) across 911 core-based statistical areas (CBSAs) sourced from Google Maps. We employ the Bidirectional Encoder Representations from Transformers (BERT) model to classify the parking sentiment and conduct regression analyses to explore its relationships with socio-spatial factors. Findings reveal significant variations in parking sentiment across POI types and CBSAs, with Restaurants showing the most negative. Regression results further indicate that denser urban areas with higher proportions of African Americans and Hispanics and lower socioeconomic status are more likely to exhibit negative parking sentiment. Interestingly, an opposite relationship between parking supply and sentiment is observed, indicating increasing supply does not necessarily improve parking experiences. Finally, our textual analysis identifies keywords associated with positive or negative sentiments and highlights disparities between urban and rural areas. Overall, this study demonstrates the potential of a novel data source and methodological framework in measuring parking sentiment, offering valuable insights that help identify hyperlocal parking issues and guide targeted parking management strategies. 

Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

arXiv, July 2024

Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky

Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts. However, recent LLMs like Gemini1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG’s significantly lower cost remains a distinct advantage. Based on this observation, we propose SELF-ROUTE, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. SELFROUTE significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.

Academic collaboration on large language model studies increases overall but varies across disciplines

arXiv, August 2024

Lingyao Li, Ly Dinh, Songhua Hu,  Libby Hemphill

Interdisciplinary collaboration is crucial for addressing complex scientific challenges. Recent advancements in large language models (LLMs) have shown significant potential in benefiting researchers across various fields. To explore the application of LLMs in scientific disciplines and their implications for interdisciplinary collaboration, we collect and analyze 50,391 papers from OpenAlex, an open-source platform for scholarly metadata. We first employ Shannon entropy to assess the diversity of collaboration in terms of authors’ institutions and departments. Our results reveal that most fields have exhibited varying degrees of increased entropy following the release of ChatGPT, with Computer Science displaying a consistent increase. Other fields such as Social Science, Decision Science, Psychology, Engineering, Health Professions, and Business, Management & Accounting have shown minor to significant increases in entropy in 2024 compared to 2023. Statistical testing further indicates that the entropy in Computer Science, Decision Science, and Engineering is significantly lower than that in health-related fields like Medicine and Biochemistry, Genetics & Molecular Biology. In addition, our network analysis based on authors’ affiliation information highlights the prominence of Computer Science, Medicine, and other Computer Science-related departments in LLM research. Regarding authors’ institutions, our analysis reveals that entities such as Stanford University, Harvard University, University College London, and Google are key players, either dominating centrality measures or playing crucial roles in connecting research networks. Overall, this study provides valuable insights into the current landscape and evolving dynamics of collaboration networks in LLM research. Our findings also suggest potential areas for fostering more diverse collaborations and highlight the need for continued research on the impact of LLMs on scientific research practices and outcomes.

The Language of Trauma: Modeling Traumatic Event Descriptions Across Domains with Explainable AI

arXiv, August 2024

Miriam Schirmer, Tobias Leemann, Gjergji Kasneci,  Jürgen Pfeffer, David Jurgens

Psychological trauma can manifest following various distressing events and is captured in diverse online contexts. However, studies traditionally focus on a single aspect of trauma, often neglecting the transferability of findings across different scenarios. We address this gap by training language models with progressing complexity on trauma-related datasets, including genocide-related court data, a Reddit dataset on post-traumatic stress disorder (PTSD), counseling conversations, and Incel forum posts. Our results show that the fine-tuned RoBERTa model excels in predicting traumatic events across domains, slightly outperforming large language models like GPT-4. Additionally, SLALOM-feature scores and conceptual explanations effectively differentiate and cluster trauma-related language, highlighting different trauma aspects and identifying sexual abuse and experiences related to death as a common traumatic event across all datasets. This transferability is crucial as it allows for the development of tools to enhance trauma detection and intervention in diverse populations and settings.

RELATED

Subscribe to our free Research Roundup newsletter today! 

— Noor Hindi, UMSI public relations specialist