Jiacheng XuHomepage

KDD 2023

ILIInternal Logical Induction for Pixel-Symbolic Reinforcement Learning

Jiacheng Xu, Chao Chen, Fuxiang Zhang, Lei Yuan, Zongzhang Zhang, and Yang Yu

Affiliations at publication: Nanjing University · Polixir Technologies

Learn from pixels.
Reason with rules.

ILI turns symbolic experience into propositional knowledge,
then uses that knowledge to guide a visual reinforcement learning agent.

ILI framework: experience replay generates a knowledge base, which provides auxiliary rewards to the agent interacting with the environment.
Rules become learning signals. Figure 4 from the KDD 2023 paper. View full-size figure ↗

01 / The method

Two observation types. Complementary roles.

Pixels provide rich perceptual information. Symbols expose compact, meaningful features. ILI connects the two through reward shaping.

Perception

Learn a visual policy.

A deep RL agent processes pixel observations. The experiments instantiate this component with a Deep Q-Network (DQN).

Induction

Extract logical knowledge.

RIPPER induces propositional rules from symbolic experience, capturing patterns from successful interactions.

Reward shaping

Use useful rules as guidance.

An adaptive mechanism selects valuable knowledge and converts it into intrinsic rewards for the visual agent.

02 / The evidence

Visual learning guided by symbolic experience.

Experiments compare ILI with pixel-only, symbolic-only, and mixed-input DQN baselines.

FlappyBird environment and learning curves: ILI compared with Pixel DQN, Symbolic DQN and Mixed DQN.CarRacing environment and learning curves: ILI compared with Pixel DQN, Symbolic DQN and Mixed DQN.CollectHealth environment and learning curves: ILI compared with Pixel DQN, Symbolic DQN and Mixed DQN.
Three pixel-symbolic environments. The learning curves show task-dependent improvements in return and sample efficiency. Figure 6 from the paper. View full-size figure ↗
FlappyBird

Rules for focused guidance.

Symbolic distances provide task-relevant information that helps guide the visual policy.

CarRacing

The representation matters.

The paper reports smaller improvements here: the symbolic state omits decision-relevant information such as speed and road corners.

Transfer

Reuse knowledge when pixels change.

Separate experiments change entity colors while keeping task logic intact. Induced symbolic knowledge helps learning in the new visual setting.

03 / Interpretation

Useful symbols are part of the setting.

ILI assumes that both pixel and symbolic observations are available. Its performance depends on the symbolic information and the quality of induced knowledge. The transfer experiments change visual semantics within the studied tasks; they do not establish general transfer to arbitrary environments.

Connections to nearby research
Neuro-symbolic reinforcement learning
ILI combines a neural deep RL policy with symbolic propositional rule induction in one learning framework.
Hybrid or multimodal observations
The task setting provides both raw visual information and compact symbolic features with explicit semantics.
Intrinsic rewards and reward shaping
Induced logical knowledge is not merely reported: selected knowledge is converted into intrinsic reward for policy learning.
Visual RL generalization and transfer
The paper studies whether learned symbolic knowledge can aid policy transfer when pixel-input semantics change.
Rule learning for sequential decision-making
Propositional rules summarize useful symbolic experience and provide a compact source of guidance for reinforcement learning.

In the authors’ words

Abstract

Reinforcement Learning (RL) has experienced rapid advancements in recent years. The widely studied RL algorithms mainly focus on a single input form, such as pixel-based image input or symbolic vector input. These two forms have different characteristics and, in many scenarios, will appear together, while few RL algorithms have studied the problems with mixed input types. Specifically, in the scenario where both pixel and symbolic inputs are available, symbolic input usually offers abstract features with specific semantics, which is more conducive to the agent’s focus. Conversely, pixel input provides more comprehensive information, enabling the agent to make well-informed decisions. Tailoring the processing approach based on the properties of these two input types can contribute to solving the problem more effectively.

To tackle the above issue, we propose an Internal Logical Induction (ILI) framework that integrates deep RL and rule learning into one system. ILI utilizes the deep RL algorithm to process the pixel input and the rule learning algorithm to induce propositional logic knowledge from symbolic input. To efficiently combine these two mechanisms, we further adopt a reward shaping technique by treating valuable knowledge as intrinsic rewards for the RL procedure. Experimental results demonstrate that the ILI framework outperforms baseline approaches in RL problems with pixel-symbolic input, and its inductive knowledge exhibits transferability advantages when pixel input semantics change.

Explore the work

Resources

Build on this work

Citation

Jiacheng Xu, Chao Chen, Fuxiang Zhang, Lei Yuan, Zongzhang Zhang, and Yang Yu. “Internal Logical Induction for Pixel-Symbolic Reinforcement Learning.” KDD 2023, pages 2825–2837. DOI: 10.1145/3580305.3599393.

@inproceedings{xu2023internal,
  title     = {Internal Logical Induction for Pixel-Symbolic
               Reinforcement Learning},
  author    = {Jiacheng Xu and Chao Chen and Fuxiang Zhang and
               Lei Yuan and Zongzhang Zhang and Yang Yu},
  booktitle = {Proceedings of the 29th ACM SIGKDD Conference on
               Knowledge Discovery and Data Mining},
  pages     = {2825--2837},
  year      = {2023},
  doi       = {10.1145/3580305.3599393}
}

Figures 4 and 6 from Jiacheng Xu et al., KDD 2023. Cropped from the conference paper; © 2023 the authors, publication rights licensed to ACM.