README.md

metadata

library_name: keras
tags:
  - GRN-VSN
  - structured-data
  - classification

Model description

This model is built using two important architectural components proposed by Bryan Lim et al. in Temporal Fusion Transformers (TFT) for Interpretable Multi-horizon Time Series Forecasting called GRN and VSN which are very useful for structured data classification tasks.

1. Gated Residual Networks(GRN) consist of skip connections and gating layers that facilitate information flow efficiently. They have the flexibility to apply non-linear processing only where needed.
2. Variable Selection Networks(VSN) help in carefully selecting the most important features from the input by getting rid of any unnecessary noisy inputs which could harm the model's performance.

Note: This model is not based on the whole TFT model but only uses the GRN and VSN components described in the mentioned paper demonstrating that GRN and VSNs on their own also can be very useful for structured data learning tasks.

Intended uses

This model can be used for binary classification task to determine whether a person makes over $500K a year.

Training and evaluation data

This model was trained using the United States Census Income Dataset provided by the UCI Machine Learning Repository. The dataset contains weighted census data extracted from 1994 and 1995 Current Population Surveys conducted by the US Census Bureau. The dataset comprises of ~300K samples with 41 input features containing 7 numerical features and 34 categorical features:

Numerical Features	Categorical Features
age	class of worker
wage per hour	industry code
capital gains	occupation code
capital losses	adjusted gross income
dividends from stocks	education
num persons worked for employer	veterans benefits
weeks worked in year	enrolled in edu inst last wk
	marital status
	major industry code
	major occupation code
	mace
	hispanic Origin
	sex
	member of a labor union
	reason for unemployment
	full or part time employment stat
	federal income tax liability
	tax filer status
	region of previous residence
	state of previous residence
	detailed household and family stat
	detailed household summary in household
	instance weight
	migration code-change in msa
	migration code-change in reg
	migration code-move within reg
	live in this house 1 year ago
	migration prev res in sunbelt
	family members under 18
	total person earnings
	country of birth father
	country of birth mother
	country of birth self
	citizenship
	total person income
	own business or self employed
	taxable income amount
	fill inc questionnaire for veteran's admin

Training procedure

0. Prepare Data: Download the data and convert the target column income_level from string to integer and finally split the data into train and validation.

1. Prepare tf.data.Dataset: Train and validation datasets created using Step 0 are passed to a function that converts the features and labels into a tf.data.Dataset for training and evaluation.

2. Define logic for Encoding input features: All features are encoded while also ensuring that they all have the same dimensionality.

   • Categorical Features: are encoded using Embedding layer provided by Keras with output dimension of embedding equal to encoding_size

   • Numerical Features: are projected into a encoding_size dimensional vector by applying a linear transformation using Dense layer provided by Keras

3. Implement the Gated Linear Unit (GLU): consists of two Dense layers where the last last dense layer has a sigmoid activation. GLUs help in suppressing inputs that are not useful for a given task.

4. Implement the Gated Residual Network:

   • Applies Non-linear ELU tranformation on its inputs
   • Applies linear transformation followed by dropout
   • Applies GLU and adds the original inputs to the output of the GLU to perform skip (residual) connection
   • Applies layer normalization and produces the output

5. Implement the Variable Selection Network:

   • Applies a Gated Residual Network (GRN) which was defined in step 4 to each feature individually.
   • Applies a GRN for the concatenation of all features followed by a softmax to produce feature weights

• Produces a weighted sum of the output of the individual GRN

6. Create Model:

• The model will have input layers corresponding to both numerical and categorical features of the given dataset
• The features received by the input layers are then encoded using the encoding logic defined in Step 2.
• The encoded features pass through the Variable Selection Network(VSN)
• The output produced by the VSN are passed through a final Dense layer with sigmoid activation to produce the final output of the model

7. Compile, Train and Evaluate Model: The model is compiled using Adam optimizer and since the model is meant to binary classification, the loss function chosen is Binary Cross Entropy. The model is trained for 20 epochs and batch_size of 265 with a callback for early stopping. The model performance is evaluated based on the accuracy and loss being observed on the validation set.

Training hyperparameters

The following hyperparameters were used during training:

Hyperparameters	Value
name	Adam
learning_rate	0.0010000000474974513
decay	0.0
beta_1	0.8999999761581421
beta_2	0.9990000128746033
epsilon	1e-07
amsgrad	False
training_precision	float32

Model Plot

View Model Plot