Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More efficient tokenization methods #8

Open
keyvank opened this issue Jun 10, 2023 · 2 comments
Open

More efficient tokenization methods #8

keyvank opened this issue Jun 10, 2023 · 2 comments
Assignees
@pcranaway
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@keyvank
Copy link
Owner

keyvank commented Jun 10, 2023

We will need to have more clever methods of tokenization in femtoGPT. Possibly, it's good to have a SentencePiece model reader.
@keyvank keyvank added help wanted Extra attention is needed enhancement New feature or request labels Jun 10, 2023
@keyvank
Copy link
Owner Author

keyvank commented Jun 10, 2023

Wanted to assign you @pcranaway to this one if you'd like.

@pcranaway
Copy link

Wanted to assign you @pcranaway to this one if you'd like.

I've been looking into it today (although I haven't worked a lot) and it seems really hard, but I'll see what I can do!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pcranaway pcranaway

Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@keyvank @pcranaway