🤗 Hugging Face Hub Client Library Cheatsheet

Quickstart 🏁

The Hugging Face Hub is the central place to discover and share ML models, datasets, and Spaces.

• Discover: Find pre-trained models and datasets 🕵️‍♀️.
• Collaborate: Share and work together on ML projects 🤝.
• Explore: Try out ML apps and demos ✨.

Installation 💻

Start by installing the huggingface_hub library:

pip install --upgrade huggingface_hub

For optional features, install extra dependencies like:

pip install 'huggingface_hub[tensorflow]'

Download Files 📥

Use hf_hub_download() to download a specific file from a repo:

from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="google/pegasus-xsum", filename="config.json")

For an entire repo, use snapshot_download():

from huggingface_hub import snapshot_download
snapshot_download(repo_id="lysandre/arxiv-nlp")

Authentication 🔐

You'll need a Hugging Face account and a User Access Token for many actions.

• Login Command:

huggingface-cli login

• Environment Variable: Set HF_TOKEN.
• Method Parameters: Pass your token as an argument.

Create a Repo ➕

Use create_repo() to create a new repository:

from huggingface_hub import HfApi
api = HfApi()
api.create_repo(repo_id="my-new-model")

Set private=True for a private repo.

Upload Files 📤

Use upload_file() for a single file:

api.upload_file(
    path_or_fileobj="/path/to/README.md",
    path_in_repo="README.md",
    repo_id="my-username/my-repo"
)

For a folder, use upload_folder().

Inference API 🧠

Run accelerated inference on Hugging Face servers with InferenceClient:

from huggingface_hub import InferenceClient
client = InferenceClient()

# Text to Image
image = client.text_to_image("A cat wearing a hat")
image.save("cat_with_hat.png")

# Chat Completion
messages = [{"role": "user", "content": "Translate 'Hello' to Spanish."}]
response = client.chat_completion(messages, model="google/flan-t5-xl")
print(response.choices[0].message.content) # ¡Hola!

Manage Repos 📁

Use the HfApi to manage your repositories:

• delete_repo(): Delete a repository.
• update_repo_visibility(): Change visibility (public/private).
• move_repo(): Rename or transfer a repo.

Discussions & Pull Requests 💬

Interact with the community using HfApi:

• get_repo_discussions(): Retrieve discussions and pull requests.
• create_discussion(), create_pull_request(): Start new discussions or PRs.
• comment_discussion(), merge_pull_request(): Participate in discussions.

Collections 📚

Organize models, datasets, Spaces, and papers with collections.

• get_collection(): Fetch a specific collection.
• list_collections(): Browse available collections.
• create_collection(), add_collection_item(): Build your own collections.

Cache Management 🧹

The huggingface_hub library caches downloaded files to speed up future access.

• huggingface-cli scan-cache: Inspect your cache usage.
• huggingface-cli delete-cache: Free up disk space by deleting cached items.

Model Cards 📝

Create informative Model Cards to describe your models.

• ModelCard.load(), ModelCard.from_template(): Create or load Model Cards.
• ModelCard.push_to_hub(): Share your Model Card on the Hub.
• metadata_update(): Update your card’s metadata.

🌟 Main Features

• Fast Training: Tokenizes 1GB of text in under 20 seconds.
• Versatile: Supports different tokenization models.
• Full Alignment Tracking: Maintains correspondence with the original text.
• Pre-Processing Included: Handles truncation, padding, and special tokens.

🚀 Quick Tour

• Build Tokenizer: Use the library to train a new tokenizer.
• Train with BPE: Uses merge rules to build tokens.
• Encode Text: Tokenize and retrieve IDs with Tokenizer.encode.
• Post-Processing: Automatically adds special tokens.

🔧 Installation

• Using Pip: pip install tokenizers.
• From Source: Requires Rust. Use git clone and compile.

🧰 Pre-Tokenizers

• Whitespace: Splits on spaces.
• Punctuation: Isolates punctuation marks.
• ByteLevel: Encodes at the byte-level.

⚙️ Models

• BPE: Merges frequent pairs of characters.
• WordPiece: Splits words into tokens.
• Unigram: Finds the most probable subword units.

🔄 Post-Processors

• TemplateProcessing: Adds special tokens for models.
• Decoders: Converts IDs back to text.

🛠️ Normalizers

• NFD: Decomposes Unicode characters.
• Lowercase: Converts text to lowercase.
• StripAccents: Removes accent marks.

🧪 Training from Memory

• Basic Method: Use any Python iterator.
• Using Datasets: Integrate with 🤗 Datasets library.
• From Gzip Files: Train directly from compressed files.

📚 Components

• Normalizers: Prepare text for tokenization.
• Pre-tokenizers: Split input text into words.
• Models: The core algorithm for tokenization.

Overview

• Transformers are state-of-the-art models for NLP, CV, and Audio tasks.
• Support for PyTorch, TensorFlow, and JAX frameworks.
• Pretrained models reduce compute costs and training time.
• Provides APIs to download and train models easily.

Tasks Supported

• NLP: Text classification, named entity recognition, question answering, text generation, etc.
• Computer Vision: Image classification, object detection, segmentation.
• Audio: Automatic speech recognition, audio classification.
• Multimodal: Table question answering, OCR, video classification, visual question answering.

Framework Interoperability

• Switch between PyTorch, TensorFlow, and JAX seamlessly.
• Export models to formats like ONNX or TorchScript for production deployment.

Getting Started

• Install: pip install transformers datasets evaluate accelerate
• Load a pretrained model: pipeline(task="task-name")
• Use AutoModel and AutoTokenizer for flexibility.

Key Pipelines

• Text Classification: pipeline(task="sentiment-analysis")
• Text Generation: pipeline(task="text-generation")
• Summarization: pipeline(task="summarization")
• Image Classification: pipeline(task="image-classification")
• Image Segmentation: pipeline(task="image-segmentation")
• Object Detection: pipeline(task="object-detection")
• Audio Classification: pipeline(task="audio-classification")
• Speech Recognition: pipeline(task="automatic-speech-recognition")
• Visual Question Answering: pipeline(task="vqa")

Auto Classes

• AutoTokenizer: Preprocess text into numerical arrays.
• AutoModel: Load a model for specific tasks (e.g., classification, generation).
• AutoConfig: Customize model architecture (e.g., number of layers, attention heads).

Model Training

• Use Trainer class for PyTorch optimized training loop.
• Use prepare_tf_dataset() for TensorFlow training with Keras API.
• Supports distributed training, mixed precision, and callbacks for custom functionality.

Installation Tips

• Install from pip: pip install transformers
• Install with dependencies: pip install 'transformers[torch]'
• Install from source: pip install git+https://github.com/huggingface/transformers
• Editable install: For contributing or using the main version.

Saving and Loading Models

• Save with save_pretrained() and reload with from_pretrained().
• Convert models between frameworks using from_pt or from_tf parameters.