대학소식 게시판목록 | Sungkyunkwan University ( SKKU ) College of Computing and Informatics

본문 바로가기
주메뉴 바로가기
서브메뉴 바로가기

High Tech, High Touch

Research

Research
Research News

Research

Research News

[Research] Three Short Papers accepted at TheWebConf (WWW) 2025 from Professor Simon S. Woo’s Lab (DASH Lab)

The Data-driven AI & Security HCI Lab (DASH Lab, Advisor: Simon S. Woo) has had three short papers accepted for publication at the International World Wide Web Conference (WWW), a top-tier international conference in BK Computer Science, covering web technologies, internet advancements, data science, and artificial intelligence. The papers will be presented in April in Sydney, Australia. 1. Towards Safe Synthetic Image Generation On the Web: A Multimodal Robust NSFW Defense and Million Scale Dataset, WWW 2025 Authors:Muhammad Shahid Muneer (Ph.D. Student, Department of Software), Simon S. Woo (Professor, Department of Software, Sungkyunkwan University) 2. Fairness and Robustness in Machine Unlearning, WWW 2025 Authors: Khoa Tran (Integrated M.S./Ph.D. Student, Department of Software), Simon S. Woo (Professor, Department of Software, Sungkyunkwan University) Machine unlearning addresses the challenge of removing the influence of specific data from a pretrained model, which is a crucial issue in privacy protection. While existing approximated unlearning techniques emphasize accuracy and time efficiency, they fail to achieve exact unlearning. In this study, we are the first to incorporate fairness and robustness into machine unlearning research. Our study analyzes the relationship between fairness and robustness based on fairness conjectures, and experimental results confirm that a larger fairness gap makes the model more vulnerable. Additionally, we demonstrate that state-of-the-art approximated unlearning methods are highly susceptible to adversarial attacks, significantly degrading model performance. Therefore, we argue that fairness-gap measurement and robustness metrics should be essential evaluation criteria for unlearning algorithms. Finally, our findings show that unlearning at the intermediate and final layers is sufficient while also improving time and memory efficiency. 3. SADRE: Saliency-Aware Diffusion Reconstruction for Effective Invisible Watermark Removal, WWW 2025 Authors: Inzamamul Alam (Ph.D. Student, Department of Software), Simon S. Woo (Professor, Department of Software, Sungkyunkwan University) To address the robustness limitations of existing watermarking techniques, this study proposes SADRE (Saliency-Aware Diffusion Reconstruction), a novel watermark removal framework. SADRE applies saliency mask-guided noise injection and diffusion-based reconstruction to preserve essential image features while effectively removing watermarks. Additionally, it adapts to varying watermark strengths through adaptive noise adjustment and ensures high-quality image restoration via a reverse diffusion process. Experimental results demonstrate that SADRE outperforms state-of-the-art watermarking techniques across key performance metrics, including PSNR, SSIM, Wasserstein Distance, and Bit Recovery Accuracy. This research establishes a theoretically robust and practically effective watermark removal solution, proving its reliability for real-world web content applications.
- 작성일 2025-03-05
- 조회수 205
[Research] One paper accepted at EuroS&P 2025 from Professor Simon S Woo's (DASH Lab)

The Data-driven AI & Security HCI Lab (DASH Lab, Advisor: Simon S. Woo) has had one System of Knowledge (SoK) paper accepted for publication at the 10th IEEE European Symposium on Security and Privacy (Euro S&P), a prestigious international conference covers Machine Learning Security, System & Network Security, Cryptographic Protocols, Data Privacy. The papers will be presented in July in Venice, Italy. SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework, EuroS&P 2025 Authors: Binh Le and Jiwon Kim (Ph.D. Student, Department of Software), Simon S. Woo (Professor, Department of Software, Sungkyunkwan University) This work is jointly performed with CSIRO Data61 as an international collaboration. Paper Link: https://arxiv.org/abs/2401.04364
- 작성일 2025-03-05
- 조회수 215
[Research] Professor Woo Hongwook’s Research Lab (CSI Lab), ICLR 2025 Paper Acceptance

[Professor Woo Hongwook’s Research Lab (CSI Lab), ICLR 2025 Paper Acceptance] Two papers from CSI Lab (Supervised by Professor Woo Hongwook) have been accepted for presentation at ICLR 2025 (The 13th International Conference on Learning Representations), a prestigious conference in the field of Artificial Intelligence. The papers will be presented in April 2025 at the Singapore Expo in Singapore. 1. Paper “Model Risk-sensitive Offline Reinforcement Learning” The author of this paper is Kwangpyo Yoo, a Ph.D. candidate in the Department of Software. This study proposes a Model Risk-sensitive Reinforcement Learning (Model Risk-sensitive RL) framework for critical mission domains, such as robotics and finance, where decision-making is crucial. The paper particularly details a model risk-sensitive offline reinforcement learning technique (MR-IQN). MR-IQN aims to minimize the "model risk" loss in cases where the model's learned data differs from the real environment, leading to decreased accuracy. To achieve this, it calculates the model's confidence in each data point and evaluates the model risk per data point using a Critic-Ensemble Criterion. It also introduces a Fourier Feature Network that limits the gap between the actual policy's value function and the inferred policy’s value in an offline setting. MR-IQN outperformed other state-of-the-art risk-sensitive reinforcement learning techniques in experiments conducted in MT-Sim (financial trading environment) and AirSim (autonomous driving simulator), achieving lower risk and higher average performance. 2. Paper “NeSyC: A Neuro-symbolic Continual Learner For Complex Embodied Tasks In Open Domains” This paper was co-authored by Wonje Choi (Ph.D. candidate, Department of Software), Jinwoo Park (Master’s student, Department of Artificial Intelligence), Sanghyun Ahn (Master’s student, Department of Software), and Daehui Lee (Integrated Master’s and Ph.D. candidate). The study proposes a Neuro-symbolic Continual Learner (NeSyC) framework that continuously generalizes knowledge (Actionable Knowledge) from embodied experiences to be applied to various tasks in open-domain physical environments. NeSyC mimics the human cognitive process of hypothesizing and deducing (hypothetico-deductive reasoning) to improve performance in open domains. This is achieved by: Using LLMs and symbolic tools to repeatedly generate and verify hypotheses from acquired experiences in a contrastive generality improvement approach. Utilizing memory-based monitoring to detect action errors of embodied agents in real-time and refine their knowledge, ultimately improving the agent's task performance and generalization across open-domain environments. NeSyC was evaluated across various benchmark environments, including ALFWorld, VirtualHome, Minecraft, RLBench, and real-world robotic tabletop scenarios. It demonstrated robust performance across dynamic open-domain environments and outperformed state-of-the-art methods, such as AutoGen, ReAct, and CLMASP, in task success rates. CSI Lab conducts research on network and cloud system optimization, autonomous driving of robots and drones, and other self-learning technologies by leveraging Embodied Agent, Reinforcement Learning, and Self-Learning. Contact Information:Professor Woo Hongwook | hwoo@skku.edu | CSI Lab | https://sites.google.com/view/csi-agent-group
- 작성일 2025-02-20
- 조회수 432
[Research] Security Engineering Lab, Two Papers Accepted at CHI 2025

[25.01.21] Security Engineering Lab, Two Papers Accepted at CHI 2025 The Security Engineering Lab (Advisor: Professor Hyungsik Kim) has had two papers accepted at CHI 2025 (ACM SIGCHI Conference on Human Factors in Computing Systems), one of the top-tier conferences in the field of Human-Computer Interaction (HCI). The papers will be presented in April 2025 in Yokohama, Japan. 1. Paper: "Understanding and Improving User Adoption and Security Awareness in Password Checkup Services" Authors: Sanghak Oh (PhD Student, Department of Electrical and Computer Engineering) Heewon Baek (MS Student, Department of Electrical and Computer Engineering) Taeyoung Kim (PhD Student, Department of Electrical and Computer Engineering) Woojin Jeon (PhD Student, Department of Electrical and Computer Engineering) Junho Heo (Samsung Research) Professor Ian Oakley (KAIST) Professor Hyungsik Kim (Sungkyunkwan University) Password Checkup Services (PCS) help users protect accounts by identifying compromised, reused, or weak passwords. However, these services have low adoption rates. This study conducted an online survey (N=238) to identify factors influencing PCS adoption and barriers to changing compromised passwords. Key findings include: Adoption factors: Perceived usefulness, ease of use, and self-efficacy were significant motivators. Barriers to password changes: Warning fatigue from frequent alerts, low awareness of password compromise risks, and reliance on other security measures discouraged users from taking action. To address these issues, the research team redesigned the PCS interface by: Clarifying warning messages related to compromised passwords. Automating the password change process, such as enabling users to update multiple reused passwords simultaneously or directly linking to password change pages. A task-based interview study (N=50) validated the effectiveness of the new design, showing a significant increase in password change rates in two scenarios: 40% and 74% change rates, compared to 16% and 60% in Google's existing PCS design. 2. Paper: "I Was Told to Install the Antivirus App, but I’m Not Sure I Need It: Understanding the Adoption, Discontinuation, and Non-Use of Smartphone Antivirus Software in South Korea" Authors: Seyoung Jin (MS Student, Department of Software) Heewon Baek (MS Student, Department of Software) Professor Euijin Lee (KAIST) Professor Hyungsik Kim (Sungkyunkwan University) This study investigates the limited effectiveness of smartphone antivirus software, despite recommendations from security firms, due to user misconceptions, regulatory requirements, and improper usage. Using a mixed-methods approach, including in-depth interviews (N=23) and a survey (N=250), the study examined the adoption status of smartphone antivirus software, particularly in South Korea, where it is often mandatory for banking and financial apps. Key findings: Many users confused antivirus software with general security tools and were unaware of its limited scope in addressing mobile malware threats. Factors influencing adoption: Perceived vulnerability, response efficacy, self-efficacy, social norms, and awareness. Factors leading to discontinuation or non-use: Concerns about system performance impact and skepticism about necessity. Additionally, the mandatory installation of antivirus software for financial apps in South Korea has contributed to user misconceptions, negative perceptions, and a false sense of security. This research highlights the need for better user education, clearer communication on mobile-specific security threats, and improved guidance to enhance effective antivirus software usage.
- 작성일 2025-02-20
- 조회수 408
[Research] CSI Lab (Prof. Hongwook Woo), Paper Accepted at AAAI 2025

[Prof. Hongwook Woo] CSI Lab , Paper Accepted at AAAI 2025 The CSI Lab (Advisor: Professor Hongwook Woo) has had its paper accepted at AAAI 2025 (The 39th Annual AAAI Conference on Artificial Intelligence), one of the prestigious conferences in the field of artificial intelligence. The paper is scheduled to be presented in February 2025 in Philadelphia, USA. Paper Details The paper, “In-Context Policy Adaptation Via Cross-Domain Skill Diffusion,” was authored by Minjong Yoo (Integrated MS/PhD Program, Department of Software) as the first author, with Wookyung Kim (Integrated MS/PhD Program, Department of Software) as a co-author. This research proposes an In-Context Policy Adaptation (ICPAD) framework for long-horizon, multi-task environments across various domains and introduces diffusion-based skill learning techniques in cross-domain settings. ICPAD is designed to rapidly adapt reinforcement learning (RL) policies to diverse target domains using only limited target domain data—without requiring model updates. To achieve this, ICPAD: Learns domain-invariant prototype skills and domain-grounded skill adapters to maintain consistency across domains while adapting policies to new target domains through cross-domain skill diffusion. Optimizes diffusion-based skill translation by utilizing limited target domain data as prompts, enhancing policy adjustment via dynamic domain prompting. Experimental Results Experiments demonstrated that ICPAD outperforms state-of-the-art (SOTA) methods in adapting to dynamic environmental changes and various domain settings in: MetaWorld (robotic manipulation environment) CARLA (autonomous driving simulator) CSI Lab Research and Funding The CSI Lab focuses on machine learning, reinforcement learning, and self-supervised learning for optimizing networks, cloud systems, robotics, and autonomous drone navigation. This AAAI 2025 research is supported by: Core AI Technology Project for Human-Centered AI (IITP) National Research Foundation of Korea (NRF) Individual Basic Research Program Graduate School of AI ICT Elite Talent Development Program BK21 FOUR Program (BK21) Institute for Information & Communications Technology Planning & Evaluation (IITP) Samsung Electronics Contact Information Professor Hongwook Woo | hwoo@skku.edu CSI Lab | https://sites.google.com/view/csi-agent-group
- 작성일 2025-02-20
- 조회수 401
[Research] LearnData Lab, Research on Graph Neural Networks Accepted at WSDM 2025

LearnData Lab's Research on Graph Neural Networks Accepted at WSDM 2025 (Master's Graduate: Jongwon Park, PhD Candidate: Heesoo Jeong) A research paper from LearnData Lab (Advisor: Professor Hogun Park) has been accepted at The 18th ACM International Conference on Web Search and Data Mining (WSDM 2025), one of the top-tier conferences in the field of artificial intelligence. Paper Details The paper, “CIMAGE: Exploiting the Conditional Independence in Masked Graph Auto-encoders”, was published with Jongwon Park (Master’s Graduate, AI Department) as the first author, and Heesoo Jeong (PhD Candidate, Software Department) as the co-first author. Research Highlights Professor Hogun Park's research team at Sungkyunkwan University has achieved significant advancements in Graph Neural Network (GNN) learning based on self-supervised learning. This study introduces a novel model called CIMAGE (Conditional Independence Aware Masked Graph Auto-Encoder), which overcomes the limitations of conventional random masking techniques and significantly enhances the expressive power of GNNs. The CIMAGE model leverages conditional independence to design a more effective masking strategy, significantly improving both efficiency and accuracy in graph representation learning. A key aspect of this research is the use of high-confidence pseudo-labels to generate two independent contexts, enabling a novel pretext task that enhances the masking and reconstruction processes. The effectiveness of CIMAGE has been demonstrated across various graph benchmark datasets, achieving outstanding performance in downstream tasks such as node classification and link prediction. This breakthrough establishes a new standard in graph representation learning. Significance and Future Applications This research represents an important milestone in Sungkyunkwan University's commitment to innovative and pioneering research. The findings have high potential for application in graph neural networks and self-supervised learning. LearnData Lab focuses on developing cutting-edge machine learning and data mining technologies across various modalities, including graphs, natural language, sensor data, and images. The lab is also actively involved in explainable AI research. The WSDM 2025 paper was supported by funding from the Graduate School of AI, the Institute for Information & Communications Technology Planning & Evaluation (IITP), and the Korea Creative Content Agency (KOCCA). Contact Information Professor Hogun Park | hogunpark@skku.edu LearnData Lab | https://learndatalab.github.io
- 작성일 2025-02-20
- 조회수 341
[Research] SALab, Papers Approved for Publication at the ICSE 2025 International Conference

[Prof. Sooyoung Cha] SALab, Papers Approved for Publication at the ICSE 2025 International Conference ■ Title: TopSeed: Learning Seed Selection Strategies for Symbolic Execution from Scratch ■ Author of a paper: Jaehyeok Lee, Prof. Sooyoung Cha ■ Conference: IEEE/ACM International Conference on Software Engineering (ICSE 2025) ■ Abstract: We present TopSeed, a new approach that automatically selects optimal seeds to enhance symbolic execution. Recently, the performance of symbolic execution has significantly improved through various state-of-the-art techniques, including search strategies and state-pruning heuristics. However, these techniques have typically demonstrated their effectiveness without considering “seeding”, which efficiently initializes program states for exploration. This paper aims to select valuable seeds from candidate inputs generated during interactions with any symbolic execution technique, without the need for a predefined seed corpus, thereby maximizing the technique's effectiveness. One major challenge is the vast number of candidates, making it difficult to identify promising seeds. To address this, we introduce a customized online learning algorithm that iteratively groups candidate inputs, ranks each group, and selects a seed from the top-ranked group based on data accumulated during symbolic execution. Experimental results on 17 open-source C programs show that TopSeed significantly enhances four distinct cutting-edge techniques, implemented on top of two symbolic executors, in terms of branch coverage and bug-finding abilities.
- 작성일 2025-02-20
- 조회수 350
[Research] A paper from Prof. Lee jinkyu's lab. Papers was approveed by publication at ACM/IEEE DAC 2024, IEEE RTAS 2024

제목: 이진규 교수 연구실(실시간 컴퓨팅 연구실, RTCL@SKKU) ACM/IEEE DAC 2024, IEEE RTAS 2024 논문 발표 실시간 컴퓨팅 연구실(지도교수: 이진규)에서 작성한 논문이 ACM/IEEE DAC 2024 (the 61th Design Automation Conference)와 IEEE RTAS 2024 (30th IEEE Real-Time and Embedded Technology and Applications Symposium)에 발표되었습니다. ACM/DAC은 Design Automation 분야의 Top1 국제 학술대회(정보과학회 최우수 등급, BK21+ IF3)이고 올해는 미국 샌프란시스코에서 2024년 6월 23일~27일 개최되었며, IEEE RTAS는 실시간 시스템 분야의 Top2 국제 학술대회(정보과학회 최우수 등급, BK21+ IF2)이며 올해는 홍콩에서 2024년 5월 13일~16일 총 29편의 논문이 발표되었습니다. ACM/IEEE DAC 2024 논문은 MCU등 소형 IoT 기기에서 인공지능 응용 작업 실행에 대한 실시간성 보장을 다루고 있으며, 실시간 컴퓨팅 연구실 석사과정 강석민 학생(제1저자), 박사과정 이성태 학생(공동제1저자), 학부과정 구현우 학생이 이진규 교수의 지도하에 참여하였고, DGIST 좌훈승 교수와의 공동연구로 진행되었습니다. IEEE RTAS 2024 논문은 메모리 부족 환경에서의 인공지능 응용 작업의 실행에 대한 실시간성 보장을 다루고 있으며, DGIST 좌훈승 교수 연구팀 주도하에 이진규 교수가 참여하였습니다. ACM/IEEE DAC 2024 홈페이지 https://www.dac.com/ IEEE RTAS 2024 홈페이지 https://2024.rtas.org/ 실시간 컴퓨팅 연구실 홈페이지 https://rtclskku.github.io/website/ - 논문제목: RT-MDM: Real-Time Scheduling Framework for Multi-DNN on MCU Using External Memory - Abstract: As the application scope of DNNs executed on microcontroller units (MCUs) extends to time-critical systems, it becomes important to ensure timing guarantees for increasing demand of DNN inferences. To this end, this paper proposes RT-MDM, the first Real-Time scheduling framework for Multiple DNN tasks executed on an MCU using external memory. Identifying execution-order dependencies among segmented DNN models and memory requirements for parallel execution subject to the dependencies, we propose (i) a segment-group-based memory management policy that achieves isolated memory usage within a segment group and sharded memory usage across different segment groups, and (ii) an intra-task scheduler specialized for the proposed policy. Implementing RT-MDM on an actual system and optimizing its parameters for DNN segmentation and segment-group mapping, we demonstrate the effectiveness of RT-MDM in accommodating more DNN tasks while providing their timing guarantees. - 논문제목: RT-Swap: Addressing GPU Memory Bottlenecks for Real-Time Multi-DNN Inference - Abstract: The increasing complexity and memory demands of Deep Neural Networks (DNNs) for real-time systems pose new significant challenges, one of which is the GPU memory capacity bottleneck, where the limited physical memory inside GPUs impedes the deployment of sophisticated DNN models. This paper presents, to the best of our knowledge, the first study of addressing the GPU memory bottleneck issues, while simultaneously ensuring the timely inference of multiple DNN tasks. We propose RT-Swap, a real-time memory management framework, that enables transparent and efficient swap scheduling of memory objects, employing the relatively larger CPU memory to extend the available GPU memory capacity, without compromising timing guarantees. We have implemented RT-Swap on top of representative machine-learning frameworks, demonstrating its effectiveness in making significantly more DNN task sets schedulable at least 72% over existing approaches even when the task sets demand up to 96.2% more memory than the GPU’s physical capacity. 이진규 | jinkyu.lee@skku.edu | 실시간컴퓨팅 Lab. | https://rtclskku.github.io/website/ Title: Papers from Prof. Jinkyu Lee’s Lab. (RTCL@SKKU) published in ACM/IEEE DAC 2024 and IEEE RTAS 2024 A paper from RTCL@SKKU (Advisor: Jinkyu Lee) has been published in ACM/IEEE DAC 2024 and IEEE RTAS 2024. ACM/IEEE DAC 2024 Website https://www.dac.com/ IEEE RTAS 2024 Website https://2024.rtas.org/ Real-Time Computing Lab. Website https://rtclskku.github.io/website/ - Paper Title: RT-MDM: Real-Time Scheduling Framework for Multi-DNN on MCU Using External Memory - Abstract: As the application scope of DNNs executed on microcontroller units (MCUs) extends to time-critical systems, it becomes important to ensure timing guarantees for increasing demand of DNN inferences. To this end, this paper proposes RT-MDM, the first Real-Time scheduling framework for Multiple DNN tasks executed on an MCU using external memory. Identifying execution-order dependencies among segmented DNN models and memory requirements for parallel execution subject to the dependencies, we propose (i) a segment-group-based memory management policy that achieves isolated memory usage within a segment group and sharded memory usage across different segment groups, and (ii) an intra-task scheduler specialized for the proposed policy. Implementing RT-MDM on an actual system and optimizing its parameters for DNN segmentation and segment-group mapping, we demonstrate the effectiveness of RT-MDM in accommodating more DNN tasks while providing their timing guarantees. - Paper Title: RT-Swap: Addressing GPU Memory Bottlenecks for Real-Time Multi-DNN Inference - Abstract: The increasing complexity and memory demands of Deep Neural Networks (DNNs) for real-time systems pose new significant challenges, one of which is the GPU memory capacity bottleneck, where the limited physical memory inside GPUs impedes the deployment of sophisticated DNN models. This paper presents, to the best of our knowledge, the first study of addressing the GPU memory bottleneck issues, while simultaneously ensuring the timely inference of multiple DNN tasks. We propose RT-Swap, a real-time memory management framework, that enables transparent and efficient swap scheduling of memory objects, employing the relatively larger CPU memory to extend the available GPU memory capacity, without compromising timing guarantees. We have implemented RT-Swap on top of representative machine-learning frameworks, demonstrating its effectiveness in making significantly more DNN task sets schedulable at least 72% over existing approaches even when the task sets demand up to 96.2% more memory than the GPU’s physical capacity. Jinkyu Lee | jinkyu.lee@skku.edu | RTCL@SKKU | https://rtclskku.github.io/website/
- 작성일 2024-07-01
- 조회수 3847
[Research] 우사이먼성일 교수(DASH 연구실), Won PAKDD 2024 Best Paper Running-Up Award (2nd Place)

DASH Lab won the Best Paper Running-Up Award (2nd Best Paper) at PAKDD 2024 in Taiwan Binh M. Le and Simon S. Woo’s paper, “SEE: Spherical Embedding Expansion for Improving Deep Metric Learning,” received the the Best Paper Running-Up Award (2nd best paper) in PAKDD 2024 (BK CS IF=1), held in Taipei in May 2024. Here is the background information about the award: “This year, PAKDD received 720 excellent submissions, and the selection process was competitive, rigorous, and thorough with over 500 PC and 100 SPC members. An award committee was formed by a chair and four committee members from different countries. There are only one Best Paper Award, two Best Paper Running-Up Awards, and one Best Student Paper Award.” Paper Link: https://link.springer.com/chapter/10.1007/978-981-97-2253-2_11
- 작성일 2024-06-07
- 조회수 4123
[Research] One paper by Intelligent Embedded Systems Laboratory (supervisor: Shin Dong-gun) has been approved for publication at AA

One paper by Intelligent Embedded Systems Laboratory (supervisor: Shin Dong-gun) has been approved for publication at AAAI Conference on Artificial Intelligence 2024 (AAAI-24), an excellent academic conference in the field of artificial intelligence 논문 #1: Proxyformer: Nystrom-Based Linear Transformer with Trainable Proxy Tokens (Lee Sang-ho, master's degree in artificial intelligence, and Lee Ha-yoon, Ph.D. in electrical, electronic, and computer engineering) The paper "Proxyformer: Nystrom-Based Linear Transformer with Trainable Proxy Tokens" focuses on the complexity of self-attention operations. In this paper, to solve the quadratic complexity of the input sequence length n of the existing self-attention operation, we propose an extended Nystrom attraction method by integrating the Nystrom method with neural memory. First, by introducing a learnable proxy token, which serves as a landmark of the Nystrom method, we reduce the complexity of the attraction operation from square to linear, and effectively create landmarks that take into account the input sequence. Second, we learn to effectively restore the attraction map using minimal landmarks by applying continuous learning. Third, we develop a dropout method suitable for the decomposed attraction matrix, enabling the normalization of the proxy tokens to be effectively learned. The proposed proxyformer effectively approximates the attention map with minimal proxy token, which outperforms existing techniques in LRA benchmarks and achieves 3.8 times higher throughput and 0.08 times lower memory usage in 4096-length input sequences compared to traditional self-attention methods. [Thesis #1 Information] Proxyformer: Nystrom-Based Linear Transformer with Trainable Proxy Tokens Sangho Lee, Hayun Lee, Dongkun Shin Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI), 2024 Transformer-based models have demonstrated remarkable performance in various domains, including natural language processing, image processing and generative modeling. The most significant contributor to the successful performance of Transformer models is the self-attention mechanism, which allows for a comprehensive understanding of the interactions between tokens in the input sequence. However, there is a well-known scalability issue, the quadratic dependency of self-attention operations on the input sequence length n, making the handling of lengthy sequences challenging. To address this limitation, there has been a surge of research on efficient transformers, aiming to alleviate the quadratic dependency on the input sequence length. Among these, the Nyströmformer, which utilizes the Nyström method to decompose the attention matrix, achieves superior performance in both accuracy and throughput. However, its landmark selection exhibits redundancy, and the model incurs computational overhead when calculating the pseudo-inverse matrix. We propose a novel Nyström method-based transformer, called Proxyformer. Unlike the traditional approach of selecting landmarks from input tokens, the Proxyformer utilizes trainable neural memory, called proxy tokens, for landmarks. By integrating contrastive learning, input injection, and a specialized dropout for the decomposed matrix, Proxyformer achieves top-tier performance for long sequence tasks in the Long Range Arena benchmark.
- 작성일 2023-12-26
- 조회수 4539