Getting data from an API

TL;DR#

Today we looked at where data comes from, read lists of dictionaries and nested replies, wrote our first functions with def, and asked a weather server for data with requests. The habit worth keeping: check what you have before you use it. Look at the outermost bracket, print the type, and check that the result is the one you meant.

Where data comes from#

The lightning talk was on web scraping. When you need data from a website, try these in order, and stop at the first one that works:

1. A download. A CSV, Excel or JSON file the site offers
2. An API. An address that hands back data in a documented format, like the Open-Meteo weather API later on this page
3. Scraping. Your code fetches the page itself and pulls out the parts you need. Beautiful Soup reads HTML in Python, and Playwright drives a real browser for pages that build themselves with JavaScript

A file you can download still comes with terms. The site's terms of use or the dataset's license say what you may do with it, and a commercial use needs more care than a class project. Europe regulates personal data through the GDPR; the United States has no single federal law of that kind.

Read the license. The MIT License lets you use, change and share code as long as you keep its copyright notice. Datasets often carry a Creative Commons license, such as CC BY, which asks you to credit the source. A repository with no license at all gives you no permission to reuse it.

To buy a domain for a site of your own, use a registrar such as Porkbun or Cloudflare.

Turning a repository into a website#

GitHub Pages serves a repository as a website, at https://<your-github-username>.github.io/<repository>/. In class, an agent turned the lightning talk's PDF into a web page, and Pages put it online:

1. On github.com, open the repository, then Settings, then Pages
2. Under Build and deployment, choose Deploy from a branch, the branch main, and a folder, then Save
3. A file named index.html in that folder becomes the home page. A page appears a minute or two after each push

The demonstration used the root folder. To publish a marimo notebook, follow Publishing a Notebook, which uses the docs folder. A repository serves one folder at a time. Anything you publish this way is public.

What we covered#

A list is read by position, a dictionary by key. orders[0] picks the first record, and orders[0]["ShipCountry"] picks one value from it. orders["ShipCountry"] stops with TypeError: list indices must be integers or slices, not str, because orders is a list. A key with the wrong capitals stops with KeyError.

In a for loop, the loop variable holds one item at a time. In for order in orders:, order is one dictionary on each pass, so order["ShipCountry"] works where orders["ShipCountry"] did not. In for stock in portfolio:, stock is one tuple, ("AAPL", 100, 173.93) on the first pass, so stock[1] * stock[2] is that holding's shares times its price. When you are unsure what the variable holds, add print(stock, type(stock)) as the first line of the loop.

A loop collects values, and set() keeps each one once. To list every ship country, start an empty list and append each order's "ShipCountry". set(countries) gives the distinct countries, and len(set(countries)) counts them.

Read nested data one bracket at a time. Look at the outermost bracket first: [ is a list, { is a dictionary. Take one step, check what you have with type(), and take the next. A list that holds a single dictionary still needs [0] before a key will work. JSON looks almost like Python; one difference is that JSON writes true and false where Python writes True and False.

Choose a structure by the shape of the data. The questions to ask are in notebook 3, section 4, and in Python §19. Rows that come out of a database often arrive as tuples, one per row; a reply from an API arrives as JSON, which becomes lists and dictionaries.

A function is written once and changed in one place. def compute_cost(portfolio): names the function and its parameter, and return hands back the result, the way a spreadsheet formula takes cells in and gives one value back. If the sales tax rate changes, you edit one line, and every call uses the new rate. Running the cell with def only stores the function; it does nothing until you call it. A docstring, the text in triple quotes under the def line, says what the function takes and returns, and it appears when you hover over a call.

Unpacking names each position. for symbol, shares, price in holdings: gives the three values of each tuple their own names, matched by position, so the loop never needs stock[1]. The number of names has to match the number of values. A star takes whatever is left: area_code, *rest = "617-555-0123".split("-") gives "617" and ["555", "0123"].

Larger programs are small functions put together. Ordering dinner in a delivery app breaks into finding a restaurant, placing the order, charging the card and sending a driver, and computing the total is a loop like compute_cost. Breaking a problem down this way is decomposition, and using a function by what it takes and returns, without reading its inside, is abstraction. When you ask an agent for a function, state its input and its output.

An API is an address that hands back data. requests.get(url) visits the address the way a browser does. .status_code says whether the server understood the request, and .json() turns the reply into Python lists and dictionaries. Open the address in Chrome first: it shows the JSON with a Pretty-print checkbox, which Safari does not. Open-Meteo's current values are refreshed every 15 minutes; the reply says so in "interval": 900, in seconds.

Read the reply as well as the status code. A latitude of 422 got 400, with a reply that says why. The misspelled Wellesly got 200 and a reply with no results in it.

Check that the first result is the place you meant. Without count=1, Open-Meteo's town search returns 10 places, and count= goes up to 100. A search for Wellesley returns places in Massachusetts, Ontario, Penang and Australia. Code that always takes [0] can answer for the wrong town; checking country or admin1 in each result lets the code choose.

Links in a downloaded notebook now open the course site. A notebook you downloaded before today may have links that open nothing when you run it on your computer. Find those pages on this site, or download the notebook again.

Public links#

• Session page
• Mini Project 1, the instructions and the options to choose from
• Mini Project 2, the instructions and the four options to choose from
• Open-Meteo documentation, every parameter of the weather API
• Publishing a Notebook, GitHub Pages for a marimo notebook
• Python handbook, Collections, for choosing a data structure

Before next session#

Next Wednesday, 10/14, will be practice and time to work on Mini Project 2.

1. Mini Project 1 is due Sunday 10/11. Choose one option, A, B, C or D. D is a tool of your own. Write your plan in sections 1 and 2 and commit it with the message mp1: plan before AI before you ask your agent anything.
2. Start Mini Project 2 once Mini Project 1 is in. It is due Sunday 10/25. Choose an option, write your plan, and commit it before you ask your agent anything. The instructions are on the Mini Project 2 page.
3. Notebook 3, sections 4 to 6, 03-collections-and-apis.py: C and D on choosing a structure, and E to G on the weather.
4. Notebook 4, sections 1 to 3, 04-functions-and-apis.py: functions, then the course API, including every page of the towns. We will do section 4, sending data to the board, together on 10/21.
5. Commit from GitHub Desktop and push as you go. Nothing counts until it is on github.com. Both notebooks are on Checkpoint 3, which closes Sunday 10/25.