Repository navigation
Forward allowed_special in Llama3Tokenizer.encode - #1121
Open
MohammadHijjawi97 wants to merge 1 commit into
Open
MohammadHijjawi97 wants to merge 1 commit into
MohammadHijjawi97 wants to merge 1 commit into
Conversation
ChatFormat.encode accepts an allowed_special argument and passes it to Llama3Tokenizer.encode, but the tokenizer swallowed it via **kwargs and called tiktoken with its defaults. As a result, text containing a Llama 3 special token (e.g. "<|eot_id|>") always raised a ValueError, even when allowed_special="all" was passed. Forward allowed_special to tiktoken (defaulting to an empty set, which keeps the current behavior when it is not given) in both the regular and the KV-cache Llama3Tokenizer, and add a test that uses a tiny byte-level tokenizer file so no download is needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ChatFormat.encode()inllms_from_scratch.llama3(andllms_from_scratch.kv_cache.llama3) takes anallowed_specialargument and passes it toLlama3Tokenizer.encode(), but the tokenizer currently drops it via**kwargsand calls tiktoken with its defaults. So text that contains a Llama 3 special token can't be encoded, even when it is explicitly allowed:Before:
After: the special token is encoded as its ID and the call works.
This PR forwards
allowed_specialto tiktoken in bothLlama3Tokenizerclasses. It defaults to an empty set, so calls that don't pass it behave as before (special tokens in plain text are still rejected).**kwargsis kept.I also added a test to
pkg/llms_from_scratch/tests/test_llama3.pythat builds a tiny byte-level tokenizer file in a temp dir (no download) and checks the default behavior,allowed_special={...}/"all", andChatFormat.encode(..., allowed_special="all")for both the regular and KV-cache variants. It fails without the change and passes with it.