AIfred: Augmented Learning through Functional Robotic Embodiment at the Desk

Currently, many learning and creative activities such as solving problems or sketching new ideas unfold on the physical desk. Research shows that students who take handwritten notes outperform those using laptops on conceptual understanding. This is because handwriting encourages reformulation on paper and screen-based tools are more prone to passive transcription [1, 2]. Despite this, current AI tools are used exclusively through screens: users type prompts and receive a textual or visual response. Then, the resulting information is transferred back to the physical workspace (e.g., drawing a picture, or writing an essay). This workflow creates a disconnect (a.k.a., context switching) where the user works on paper while guidance lives on a screen.

Spatial Augmented Reality (SAR) (i.e., a technique that projects virtual images onto real objects) offers a way to close this gap by embedding digital information directly into physical workspaces [3], [4], [5]. More recently, robotic projection platforms like LuminAR  [6] and ELEGNT [7] introduced a robotic desk companion that enables interactive projection on arbitrary surfaces. However, these systems primarily use robotic embodiment to reposition projected content or to communicate through expressive motion. They do not perceive the user’s physical task, generate context-dependent learning support, and project that guidance back into the workspace where the task is being performed.

In this work, we present AIfred, a desk-based robotic arm with a mini projector mounted at its end-effector. An overhead camera captures the user’s workspace, while a motion-capture system tracks an object that the user moves to indicate where projected content should appear (see Fig. 1). This setup allows the system to reposition AI-generated content anywhere on the desk, keeping digital guidance physically aligned with the user’s handwritten work.

AIfred operates through a spatial assistance pipeline comprising three stages: i) workspace perception: in which an overhead camera captures the physical task context and detects the user’s pointing gesture, ii) context-aware content generation: in which a multimodal AI model interprets the captured scene and produces task-relevant guidance and iii) robot-mediated projection: in which the robotic arm projects the generated content alongside the user’s physical task.

Through the different interaction modes (i.e., math homework, generate image, and draw), we demonstrate a range of desk-based learning and creative activities. Our work investigates whether embedding AI guidance directly in the physical workspace through AIfred improves short term learning transfer, creative output quality, and user experience compared to conventional screen-based AI assistance. We additionally measure context-switching behavior and task engagement to understand the mechanisms through which spatial co-location may produce these effects.

To evaluate whether AIfred offers measurable benefits over screen-based alternatives, we conducted a user study comparing AIfred against ChatGPT (GPT-5.6 Luna) running on a laptop. Participants ($n=36$) were tasked to complete math-related assignments, image-generation from sketches, and drawing tasks (i.e., the different interaction modes) using both approaches. We measured outcome quality, short-term learning transfer, task completion time, and observed physical-digital context switches. Participants also rated their subjective experience such as perceived learning support, cognitive demand, perceived context switching, and perceived productivity (see Sec. IV).

Our results show that AIfred’s benefits concentrate in short-term learning transfer and drawing quality. In the math assignment, the two conditions performed comparably while assistance was available (6.7/10 with AIfred vs. 7.3/10 with ChatGPT, $p=.41$), but once assistance was removed the ChatGPT group fell to 4.4/10 while the AIfred group held at 7.0/10, a 60% higher short-term transfer score ($p=.003$). In the drawing task, three independent art and design professors ranked drawings produced with AIfred best in 92% of cases, with qualitative comparisons showing gains in overall form, proportions, and line quality. Participants also reported higher perceived learning support ($p<.001$) and higher cognitive demand ($p=.03$) with AIfred, with no accompanying loss in perceived productivity ($p=.61$). We hope these findings inspire future research on embodied AI systems that integrate AI assistance directly into physical learning and creative activities.

Robots are increasingly entering educational settings as tutors, companions, and collaborative learners. The physical presence of robots in learning environments has been shown to influence student engagement, motivation, and learning outcomes in ways distinct from purely screen-based systems [8, 9]. Educational robots span a range of embodiments, from humanoid social robots like NAO [10, 11], Jibo [12] and Pepper [13], to zoomorphic companions [14] and desktop robots used for support in specific learning activities [15]. Much of this research focuses on the robot as a tutor that provides feedback, demonstrates empathy, adapts its tutoring strategies, or serves as a peer learner [16, 17]. In this realm, studies show that robots capable of expressing social cues through gaze, gesture, and movement can provoke higher engagement than screen-based agents [8, 18]. This social dimension is particularly important for long-term interaction, where repeated exposure may reduce novelty effects and require more sophisticated behavioral design [19].

Beyond tutoring, recent works explore robot manipulators as demonstrators in STEM education. Projects like CoWriter [20] uses robots to apply the learning-by-teaching paradigm, where children correct a “poorly writing” robot, thereby reflecting on their own letter formation. Other work investigates how robots can physically demonstrate scientific principles, assist in assembly tasks, or act as tangible programming interfaces [21]. These systems illustrate how embodiment can move beyond social expression toward functional support that leverages the robot’s physical capabilities. However, most educational robots have been studied primarily as tutors or learning companions that provide feedback, explanations, or social interaction, rather than as systems that augment the learner’s ongoing physical work [9, 8]. While prior research has explored physical-material perception and projection-based assistance, these capabilities have generally been investigated separately from adaptive, AI-driven interpretation of the learner’s workspace [4, 22]. In addition, previous works require users to shift attention to the robot as a separate entity. This interaction model positions the robot as a tutor standing beside the user or as a companion sharing the space, rather than as an integrated assistant embedded within the users workspace [23].

The integration of digital information into physical environments has been a central theme in human-computer interaction research for over three decades. SAR extends augmented reality by embedding digital content directly into the environment through projection [3]. DigitalDesk pioneered the concept of a projector-camera desk that could recognize physical documents, augment them with digital content, and respond to finger-based interaction [4]. Subsequent systems such as metaDESK [24] and Urp [25] extended this idea by integrating tangible user interfaces that users could manipulate while digital projections updated in real time. These systems illustrated how spatial computing can allow users to think and act with physical objects while digital information remained co-located on the same surface.

More recent work has explored robot-mediated projection system that can dynamically reposition digital content across a workspace. LuminAR [6] and ELEGNT [7] represent the two systems whose design is most similar to ours (AIfred). LuminAR introduced a robotic desk lamp equipped with a projector and camera, enabling projection and touch interaction on arbitrary surfaces within reach. The system allowed users to move the lamp to project content wherever needed, blending the mobility of a physical lamp with interactive projection capabilities. Building on this lamp-like form factor, ELEGNT integrated a 6-DOF robotic arm with a projector, camera, and light to explore how expressive movement design can enhance human-robot interaction in domestic settings. While both LuminAR and ELEGNT demonstrate the potential of robotic projection platforms, LuminAR focuses on manual repositioning for interactive projection, and ELEGNT focuses on expressive movement as a communication channel and explores a broad design space for home interaction. Neither system addresses real-time task interpretation through AI or embeds learning support into ongoing desk work.

Projection-based interfaces have also been applied specifically to learning contexts. Systems such as Augmented Studio and Assemblé project instructional overlays onto physical materials to support craft, assembly, and making tasks  [26, 27]. These systems demonstrate that on-site projection can reduce the cognitive cost of switching between instructions and action. However, they typically rely on predefined content, pre-registered materials, or scripted tutorials, and do not interpret or adapt to evolving work. AIfred, in contrast, generates task-relevant guidance through generative AI in real time, directly augmenting the user’s outcomes, performance, and learning as the task unfolds.

AIfred’s prototype ¹¹ 1 All code and materials used in this work is publicly at: https://github.com/IERoboticsAILab/AIfred. (Fig. 2 A) is composed of a robotic arm (A.1) with a mini projector attached at its end-effector (A.2), which is integrated in the user workspace (Fig. 2 B). The robotic arm is placed at the edge of the desk so that its end-effector can reach and orient toward the user’s desk. To know where information should be projected, the system uses an OptiTrack motion-capture system (B.1) to track the pose of the robot base and a trackable object placed on the desk (B.2). An inverse kinematics solver then computes the arm configuration required to place the mounted projector toward the target region while maintaining a fixed projection height over the desk. The tracked object allows the user to reposition the projected content to wherever it is most relevant on the desk. Complementarily, an overhead camera (B.3) provides a top-down view of the workspace and captures the physical task context, along with the user’s pointing gesture. These visual inputs are used by the software stack to infer the user’s task context and generate the corresponding projected output (B.4).

We used a mixed experimental design to compare AIfred with ChatGPT, which we selected as the screen-based baseline because it is the AI tool students most commonly use for academic work [28]. Participants were assigned to one of two conditions: one group completed the tasks using ChatGPT on a laptop, while the other completed the same tasks using AIfred. This design allowed us to compare the two systems while keeping task categories consistent across conditions.

We designed three main desk-based task categories: math assignment, image generation, and drawing. In the math-assignment task, participants practiced problem solving on a quadratic equation, selecting from a pool of five equations of comparable difficulty to ensure equivalence across participants and conditions. In the image-generation task, participants practice rendering a physical sketch (any visual idea they drew on a paper) into a digital image. In the drawing task, users practice drawing a chosen figure among a car, a motorbike, a lion or an elephant.

The experiment began with participants performing a math assignment and an image generation task with the assistance of the assigned system (AIfred or ChatGPT). Participants then performed three drawing tasks in sequence: the first one with no assistance, the second with the assistance of their assigned system and the third with the assistance of the opposite system, as part of a crossover trial. This means that participants in the ChatGPT condition used AIfred for the third drawing task, while participants in the AIfred condition used ChatGPT. This crossover was included to enable direct within-participant comparisons of drawing outcomes across systems. Finally, participants completed an additional math assignment, solving another quadratic-equation exercise without any assistance. This task was included to assess short-term learning transfer. This metric allows us to examine whether participants could still apply relevant reasoning after the assistance was removed, with a time distance between the two math assignment of on average thirty-five minutes.

In this paper, we investigated whether delivering AI assistance directly into the physical workspace changes how users learn and create during desk-based activities. Both behavioral and self-reported findings show that AIfred improves the character of the interaction compared to ChatGPT. Participants working with AIfred sustained their engagement on paper, produced drawings that art and design professors preferred in the large majority of cases, and retained more of what they had practiced once the assistance was withdrawn. We suspect screen-based guidance often leads to passive transcription, as users can copy content with minimal reformulation. By contrast, projected guidance requires users to actively reinterpret and reproduce information by hand, promoting deeper task engagement.

The results also reveal that the benefit of spatial co-location varies across tasks. For activities such as drawing, where the user must continuously compare a reference against the proportions, angles and placement of their own lines, projected assistance was preferred. For activities such as solving a quadratic equation, AIfred and ChatGPT performed comparably while assistance was available ($6.7/10$ vs. $7.3/10$, $p=.41$). This means that the task types matter. Symbolic content can be read from a screen, held briefly in memory, and reproduced on paper. In contrast, spatial guidance relies heavily on direct visual alignment with the physical work. Every context switch to a screen breaks this visual reference, forcing the user to mentally re-align the guidance onto their paper upon looking back. As a result, co-locating guidance matters most when the instructions and the task share a unified spatial frame. Our findings imply that robot-mediated projection is not a general-purpose replacement for screen-based assistance, but a targeted one.

Finally, participants using ChatGPT performed 63 observed physical-digital context switches, compared with 1 with AIfred, yet both groups rated the disruption as low ($1.7/5$ and $1.3/5$). This indicates that participants did not consciously register screen-to-paper switching as disruptive, even though it occurred frequently and coincided with less favorable outcomes, such as lower short-term transfer and lower drawing quality. We interpret this as evidence that attentional fragmentation can operate below the threshold of subjective awareness. Users may be sufficiently accustomed to alternating between physical and digital media that the cost of switching goes unnoticed, even as it quietly shapes performance.

In this paper, we present AIfred, a system for delivering generative AI assistance directly within the user’s physical workspace. We use a desk-based robotic arm with a projector mounted on the end-effector to develop three interaction modes: math homework, image generation, and drawing. We conduct a user study with 36 participants to compare AIfred against a screen-based AI assistant across these desk-based tasks. Our results indicate that spatially co-located assistance improves drawing quality, with art and design professors ranking AIfred drawings first in 33 of 36 cases. Short-term learning transfer was also improved, with participants scoring 2.6 points higher on a 10-point scale. Behavioral analysis further shows that participants performed 63 physical-digital context switches with the screen-based assistant and 1 with AIfred, yet rarely registered this switching as disruptive, suggesting that the cost of attentional fragmentation is largely invisible. Our findings position systems such as AIfred as a promising direction for AI-assisted learning tools that support understanding.

AI was used during the preparation of this manuscript for grammar and language editing, literature research, code assistance, and data visualization/plot generation. All research decisions, analyses, and final content were reviewed and verified by the authors.