403 — Consuming JSON REST APIs

Intermediate

Go beyond single API calls: parse JSON responses into Python data structures, navigate nested and optional fields safely, paginate through multi-page GitHub API results using the Link header, and implement explicit exponential backoff for rate-limited endpoints. The lesson closes with a reusable json_api_client.py that imports check_credentials() from lesson 402 and composes all these patterns into a single, production-ready function.

Learning Objectives

Call response.json() to parse an API response into a Python dict and access top-level fields

Navigate nested fields and optional keys safely with .get() to avoid KeyError on missing data

Extract specific fields from a list response by iterating over JSON arrays

Detect pagination from the GitHub Link header and extract the rel="next" URL

Collect all pages of a paginated endpoint in a loop with a max-pages safety limit

Implement explicit exponential backoff for 403 rate-limit and 429 responses using time.sleep()

Build a reusable get_all_pages() function that combines pagination with retry logic

Step 1

Parse a JSON API response into a Python dict

Call the GitHub /repos endpoint and use response.json() to convert the response body into a Python dict. Access top-level string, integer, and boolean fields. Print formatted output from the parsed data.

Commands to Run

mkdir -p ~/devops-python/lesson-403

cd ~/devops-python/lesson-403

source ~/devops-python/lesson-101/devops-env/bin/activate

cp ~/devops-python/lesson-402/.env ~/devops-python/lesson-403/.env

cp ~/devops-python/lesson-402/config.yaml ~/devops-python/lesson-403/config.yaml

cp ~/devops-python/lesson-402/.gitignore ~/devops-python/lesson-403/.gitignore

Files for This Step

parse_response.pyPaste this into the file

import os
import sys
from pathlib import Path

import requests
from dotenv import load_dotenv

sys.path.insert(0, str(Path.home() / 'devops-python' / 'lesson-304'))
from layered_config import load_config

load_dotenv()
config, _ = load_config(
    config_file=Path('config.yaml'),
    env_file=Path('.env'),
)

base = config.api_base_url
timeout = int(config.api_timeout)
token = os.environ.get('GITHUB_TOKEN', '')
headers = {
    'Authorization': f'token {token}',
    'Accept': 'application/vnd.github+json',
    'X-GitHub-Api-Version': '2022-11-28',
}

print('=== Parsing a JSON API response: GET /repos/python/cpython ===')
r = requests.get(f'{base}/repos/python/cpython', headers=headers, timeout=timeout)
r.raise_for_status()

# response.json() parses the JSON body and returns a Python dict
repo = r.json()
print(f'Type of response.json(): {type(repo).__name__}')
print()

# Access top-level fields by key
print(f'full_name:          {repo["full_name"]}')
print(f'description:        {repo["description"]}')
print(f'language:           {repo["language"]}')
print(f'default_branch:     {repo["default_branch"]}')
print(f'stargazers_count:   {repo["stargazers_count"]:,}')
print(f'open_issues_count:  {repo["open_issues_count"]:,}')
print(f'forks_count:        {repo["forks_count"]:,}')
print(f'private:            {repo["private"]}')
print(f'archived:           {repo["archived"]}')
print()

# Show the total key count so learners understand the full response is larger
print(f'Total top-level keys in response: {len(repo)}')
print('(Printing selected fields only — use print(repo.keys()) to explore all)')
print()
print('Content-Type header:', r.headers.get('Content-Type', 'n/a'))
print('Status code:', r.status_code)

After Saving

python3 parse_response.py

What This Does

The requests.Response.json() method calls json.loads() on the response body and returns a Python object — a dict for JSON objects, a list for JSON arrays, or a scalar for bare JSON values.

The GitHub /repos endpoint returns a JSON object (dict) with over 100 keys describing the repository.

You access fields with the same dict syntax you would use for any Python dict: repo['key'].

The Content-Type: application/json header confirms the response body is JSON before .json() is called — requests will raise json.JSONDecodeError if the body is not valid JSON, so checking the status code first (via raise_for_status()) ensures you are not trying to parse an error HTML page.

Expected Outcome

Output shows Type of response.json(): dict, followed by the cpython repository's name, description, language (Python — GitHub's primary-language detection weights file bytes; cpython's large Python stdlib now outweighs its C source), default branch (main), star count (formatted with commas), open issues count, forks count, private status (False), and archived status (False).

The total key count line shows a number greater than 80.

Pro Tips

1
Use `print(list(repo.keys()))` to explore all top-level fields in a response you have not seen before. For deeply nested responses, `import json; print(json.dumps(repo, indent=2, default=str))` pretty-prints the full structure.
2
The `:,` format specifier in f-strings adds comma separators to integers — `f'{42000:,}'` produces `'42,000'`. Useful for star counts and issue counts that can be in the thousands.

Common Mistakes to Avoid

⚠️Calling `response.json()` before checking the status code — if the server returns a 4xx or 5xx error, the body may be an HTML error page that `json()` will fail to parse with a JSONDecodeError. Always call `raise_for_status()` before `json()`.
⚠️Assuming the response is always a dict — some endpoints return a JSON array (list) as the top-level value. The GitHub /repos/{owner}/{repo}/commits endpoint returns a list. Use `type(response.json()).__name__` to confirm the structure before accessing keys.

Was this step helpful?