Improve tokenizer pretty-pretty logic + __call__ method #1417
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request was exported from Phabricator. Differential Revision: D64998804 |
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 25, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 25, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
force-pushed
the
export-D64998804
branch
from
f4b7a1f to
a5ac2cd
Compare
|
This pull request was exported from Phabricator. Differential Revision: D64998804 |
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 25, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 25, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 25, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
force-pushed
the
export-D64998804
branch
from
a5ac2cd to
bbc6653
Compare
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
|
This pull request was exported from Phabricator. Differential Revision: D64998804 |
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
Summary: Slight bug in the retry GitHub workflow causes failing workflows to rerun indefinitely. This _should_ fix it Differential Revision: D65008676
Summary: Add __call__ to TokenizerLike for transformers compatibility Differential Revision: D64998805
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
force-pushed
the
export-D64998804
branch
from
bbc6653 to
6433757
Compare
|
This pull request was exported from Phabricator. Differential Revision: D64998804 |
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
craymichael
added a commit
to craymichael/captum
that referenced
this pull request
Oct 26, 2024
Summary: Use the __call__ method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string. Differential Revision: D64998804
|
This pull request has been merged in ad89e0b. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Use the call method of tokenizers that returns a BatchEncoding with offsets. This allows us to grab text from the fully decoded string and not make assumptions about how many tokens correspond to a single string.
Differential Revision: D64998804