Structure-Aware Reinforcement Learning

About Me

Education

BS Computer Engineering	Virginia Tech
BS Mathematics	Virginia Tech
MS Computer Engineering	Virginia Tech

Experience

• Chemistry ML (predicting solvation in proteins)
• Introduction to Python Instructor (HCPS)

Interests

• Swimming
• Woodworking

Nutmeg

Motivating Example

Packet Routing

• stationary environment
  transitions are deterministic for this example
  rewards (routing traffic) can be enabled/disabled to observe ideal vs actual performance
• UNISOFT features
  since packets spawn randomly, the features of the initial conditions span the feature space
• structure dependent strategy
  networks have a strategy inherit to their structure
  the models is unable to directly observe the network, and can only infer structure through trial and error

Toy Environment

A packet routing toy for learning to navigate networks
It consists of:

• Towers
  • Position (lat, lon)
• Links
  • Bandwidth (packets wait in queues to be transmitted)
  • Latency (cost)
• Packets
  • Source
  • Destination

Exploitable Structures

networks whose structures suggest a specific strategy, that an unaware agent might be slow to exploit

• k-nearest neighbors

  simple, default network structure

• big-small world

  locally dense - globally close

• directed ring

  packets flow in one direction, 1-2 steps at a time
  simple strategy that should be easy to exploit

• bottleneck

  tempting short path that is constantly congested

• parallel highways

  two parallel highways with sparse connections

Structured Problems

input features occupy a much smaller portion of the overall feature space, making it possible to assume specific properties

• Convolutional networks

  with images and audio, nearby information is relevant, and offset invariance is important, so small kernels can be re-used across the input features.

• Recurrent networks

  with sequential data, past information is relevant, and temporal dependencies are important, so recurrent connections can be used to capture these dependencies.

• RL MDPs

  purely state dependant tasks can make assumptions about advantage estimates from previous experiences

• Cross Attention

  assuming that visual / textual tasks rely on granular attention within themselves, but only semantic attention between mediums allows for more efficient

Metric for Structure Exploitation

Feature Embeddings

H scores can estimate the generalizability of embeddings from one task to another by minimizing feature redundancy and maximizing class feature discrimination

it is important to keep feature mutual information small, but instead of maximizing class relevance for transfer, structure exploitation should be independent of class labels

Designing Custom Models

• Identify structures in data
• use these to make assumptions about relevant features
• design core embeddings that conform to these assumptions