Challenge: Implementing Multi-Head Attention
Swipe um das Menü anzuzeigen
Task
You have all the building blocks: scaled dot-product attention from the previous challenge, and the intuition behind multiple heads from the last chapter. Now put them together.
Implement a MultiHeadAttention module as an nn.Module class. It should:
- Accept
d_modelandnum_headsas constructor arguments – assert thatd_model % num_heads == 0; - Define separate linear projections for
Q,K,V, and a final output projection; - In
forward(x), split the projections intonum_headsheads of dimensiond_model // num_heads; - Run scaled dot-product attention independently per head;
- Concatenate the head outputs and pass through the output projection.
Implement the module locally.
Danke für Ihr Feedback!
Fragen Sie AI
Fragen Sie AI
Fragen Sie alles oder probieren Sie eine der vorgeschlagenen Fragen, um unser Gespräch zu beginnen
Challenge: Implementing Multi-Head Attention
Task
You have all the building blocks: scaled dot-product attention from the previous challenge, and the intuition behind multiple heads from the last chapter. Now put them together.
Implement a MultiHeadAttention module as an nn.Module class. It should:
- Accept
d_modelandnum_headsas constructor arguments – assert thatd_model % num_heads == 0; - Define separate linear projections for
Q,K,V, and a final output projection; - In
forward(x), split the projections intonum_headsheads of dimensiond_model // num_heads; - Run scaled dot-product attention independently per head;
- Concatenate the head outputs and pass through the output projection.
Implement the module locally.
Danke für Ihr Feedback!